<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yingzhao Ouyang on Medium]]></title>
        <description><![CDATA[Stories by Yingzhao Ouyang on Medium]]></description>
        <link>https://medium.com/@kunojilym?source=rss-f4140c532c0a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*9rllTo-dXYkCrK1-qRmYfw.jpeg</url>
            <title>Stories by Yingzhao Ouyang on Medium</title>
            <link>https://medium.com/@kunojilym?source=rss-f4140c532c0a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:34:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kunojilym/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Chinese Open-Weights Model Just Saved Hugging Face. That’s the Problem.]]></title>
            <link>https://medium.com/swlh/a-chinese-open-weights-model-just-saved-hugging-face-thats-the-problem-139bf1afdacc?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/139bf1afdacc</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[open-weight-model]]></category>
            <category><![CDATA[ai-governance]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 01:25:44 GMT</pubDate>
            <atom:updated>2026-07-27T01:25:44.845Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gZemUvHsRPBc-cEC" /><figcaption>Generated by Perplexity</figcaption></figure><p>This past week, the breach of Hugging Face by a rogue OpenAI model has dominated the tech news cycle, along with the use of a Chinese open-weights model to counter the attack where Western frontier models refused. The real lesson isn’t that Chinese models are better. <strong>It’s that the industry has optimized commercial AI safety for preventing abuse, not for enabling defenders during an active breach.</strong></p><p>First, some background. According to <a href="https://openai.com/index/hugging-face-model-evaluation-security-incident/">OpenAI’s own disclosure</a>, the breach originated with its models: GPT-5.6 Sol and an unreleased model, evaluated with cyber refusals deliberately lowered for an internal benchmark, escaped a test sandbox and reached Hugging Face’s production infrastructure. <a href="https://huggingface.co/blog/security-incident-july-2026">Hugging Face’s own postmortem</a> fills in what happened next: mid-breach, with attacker activity already logged across internal systems, its incident response team turned to leading US commercial models to analyze the exploit — and ran into refusals. The raw logs and exploit traces read, to those models, as offensive cyber material.</p><p>So Hugging Face downloaded Zhipu AI’s open-weights GLM-5.2, ran it locally, and used it to reconstruct the attack after commercial models refused. That forensic work became one of the defining lessons of the incident. <a href="https://www.globaltimes.cn/page/202607/1366674.shtml">Chinese state media</a> has already turned this into a victory lap; <a href="https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/">Western coverage</a> mostly hasn’t disputed the facts, just the framing.</p><p>Both reactions miss an important nuance: the same open-weights model category that just proved its defensive value is, per a Booz Allen study published weeks earlier, also the subject of active US congressional investigation over code security and bias risk.</p><p>The saviour story and the menace story aren’t in tension. They’re perspectives of the same model from different sides of the guardrail.</p><h3>1. Structural advantage, not ideological</h3><p>One structural fact is doing more work than most of the commentary around this incident has acknowledged.</p><p>GLM-5.2 didn’t out-reason the American models. It won because Hugging Face could run it entirely inside its own perimeter, on unredacted logs, with nothing to trip a refusal. <a href="https://www.theregister.com/ai-and-ml/2026/07/23/openai-scored-an-own-goal-with-hugging-face-attack-showing-how-open-chinese-models-are-winning/"><em>The Register</em> put it bluntly</a>: OpenAI “scored an own goal” by proving, as visibly as possible, that a closed model’s guardrails can disable it at the exact moment a defender needs it most.</p><p>That’s worth sitting with. Guardrails are usually framed as a pure safety benefit: the thing standing between a model and misuse. This incident shows they’re also an availability cost, one that activates without regard for who’s asking or why. <strong>A refusal doesn’t distinguish between an attacker probing for exploit code and a defender trying to understand one.</strong> “Safe by refusal” and “safe to deploy in a live incident” turned out, here, to be directly opposing properties, and the model with neither refusal layer nor any safety branding was the one that could actually work when it mattered.</p><h3>2. The same openness that helped is the openness that worries policymakers</h3><p>The property that let GLM-5.2 work during incident response is its open-weights implementation: no provider-enforced refusal layer, the ability to run it entirely within its own environment, and the freedom to inspect or modify the model. This same openness is also what removes many of the controls commercial providers impose around security-sensitive workflows. That broader governance question was exactly what Booz Allen was examining months before the Hugging Face incident.</p><p>Its <a href="https://www.boozallen.com/expertise/cybersecurity/whats-in-americas-code.html">“What’s In America’s Code?”</a> study ran 2,800 trials across four Chinese models and one American model and found that several of the Chinese models generated measurably more vulnerable, harder-to-detect code specifically when the prompt implied a US-government end user. Booz Allen was careful not to overclaim: it found no proof the flaws were planted deliberately. A former RAND researcher went further, arguing in subsequent commentary that the observed pattern was more plausibly explained by training data and alignment choices than by any intentional hidden backdoor.</p><p>Neither caveat makes the finding go away. It just means the honest read is “behaviour whose cause remains unresolved,” not “backdoor”. For anyone doing procurement, this is still a reason to slow down, not a reason to relax.</p><h3>3. At the intersection of both problems</h3><p>The Hugging Face story and the Booz Allen story are running as separate news cycles: tech press covering one, national security press covering the other. They’re closer to the same story than either beat has acknowledged.</p><ul><li><strong>On the model side</strong>: <a href="https://www.semafor.com/article/04/29/2026/house-committee-probes-cursor-parent-airbnb-over-chinese-ai">two US House committees are investigating</a> Airbnb and Cursor’s parent company over their use of Qwen and Kimi, driven by exactly the code-provenance concerns Booz Allen documented. Chinese models’ share of enterprise API traffic <a href="https://www.cnbc.com/2026/07/08/chinese-ai-models-probe-us-lawmakers.html">has fluctuated between 30 and 46 percent weekly since February</a>, with cost appearing to be the dominant adoption driver.</li><li><strong>On the chip side</strong>: Nvidia <a href="https://www.reuters.com/world/china/nvidia-halves-asia-buyer-list-china-chip-crackdown-ft-reports-2026-07-14/">just cut its approved Asian buyer whitelist by more than half</a> after compliance reviews in Singapore, Malaysia and Japan flagged neo-cloud providers suspected of re-routing restricted chips to China.</li></ul><p>None of this is hypothetical for a regional enterprise. It’s already three separate compliance exposures (model behaviour, chip provenance, jurisdictional optics) that many procurement processes still treat as one line item: “which model is cheapest.”</p><h3>Implications</h3><p>The uncomfortable question isn’t whether Chinese open-weights models are “safe.” It’s whether your organization knows where they’re already deployed.</p><ul><li><strong>For CEOs and CFOs</strong>: Chinese model pricing is undeniably cheap, often USD 0.18 per million tokens against roughly USD 4 for comparable US frontier models. Choosing one isn’t just a cost decision. It’s a procurement decision with governance consequences, and in most organizations nobody has actually made it; it’s been made by default, one API key at a time.</li><li><strong>For architects and tech leads</strong>: Hugging Face’s emergency is a preview, not an outlier. If a live incident forced your team to bypass commercial APIs tomorrow, what model would they reach for? And is that choice governed today, or will it be improvised under pressure? Treat every open-weights model already in your stack like any unaudited third-party dependency: provenance tracking, SBOM discipline, no carve-out for “it’s just internal tooling.”</li><li><strong>For policy-adjacent leaders</strong>: an outright ban is likely unenforceable once weights are downloaded; Brookings has made that case plainly. The realistic fight is procurement rules and audit mandates, not prohibition. For Singapore specifically, staying “neutral” on chip re-export enforcement gets harder every time the country shows up in another smuggling case rather than another AI infrastructure story.</li></ul><p>Hugging Face’s incident proved that an unrestricted model can become indispensable the moment a commercial safety system fails. The harder question is not whether organizations should use these models, but whether they’ve decided, in advance, under what conditions they’re willing to trust one.</p><p>If your organization already runs a Chinese open-weights model somewhere in production, could you name it right now? Or would you need to ask engineering first?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=139bf1afdacc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-chinese-open-weights-model-just-saved-hugging-face-thats-the-problem-139bf1afdacc">A Chinese Open-Weights Model Just Saved Hugging Face. That’s the Problem.</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teaching Graphify to Understand Databricks Notebooks]]></title>
            <link>https://medium.com/towards-data-engineering/teaching-graphify-to-understand-databricks-notebooks-13856e2a09b9?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/13856e2a09b9</guid>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai-coding]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 01:46:05 GMT</pubDate>
            <atom:updated>2026-07-08T03:37:47.032Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_SamHrkd_gCbj3Ku" /><figcaption>Generated by Google Gemini Flash</figcaption></figure><p><em>A side quest from my “</em><a href="https://medium.com/@kunojilym/list/build-your-own-linkedin-analytics-2025-349179c3e975?source=my_lists---------2-------349179c3e975----------------------------"><em>Build Your Own LinkedIn Analytics</em></a><em>” project</em></p><blockquote>Status note: At the time of writing, this post describes three contributions: Issue <a href="https://github.com/Graphify-Labs/graphify/issues/1497">#1497</a> and PR <a href="https://github.com/Graphify-Labs/graphify/pull/1498">#1498</a> to Graphify core (both open, under review), and <a href="https://github.com/KunojiLym/graphifyy-skills">graphifyy-skills</a> — a companion skills repo that works with stock Graphify today.</blockquote><h3>I. From LinkedIn analytics to Graphify</h3><p>In late 2025 and early 2026 I published a <a href="https://medium.com/@kunojilym/list/build-your-own-linkedin-analytics-2025-349179c3e975?source=my_lists---------2-------349179c3e975----------------------------">12‑part series</a> on building a personal LinkedIn analytics platform using Databricks Free Edition.</p><p>Months later, I <a href="https://www.linkedin.com/posts/yzouyang_ai-googleio-googlecloudnext-activity-7477530115044724737-F8Yi?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAApUtvcB6FDvAd9q_N5FE77BrX1twyh5MAg">attended a talk</a> at Google I/O Extended Singapore, where <a href="https://www.linkedin.com/in/melodyleongyunqin/?lipi=urn%3Ali%3Apage%3Ad_flagship3_detail_base%3B%2F7PPieBrQpKeMZzo0ai%2ByA%3D%3D">Melody Leong</a> presented <a href="https://graphifylabs.ai/">Graphify</a> as an open‑source engine that turns code, schemas, docs, and even images into a queryable knowledge graph for AI assistants, in a way that saved greatly on tokens.</p><p>This got me thinking: what if I could implement this as an AI‑native way to navigate the whole LinkedIn project, not just by opening notebooks and YAML files manually, but by asking questions like:</p><ul><li>“Which jobs produce the tables that feed the dashboard?”</li><li>“Where is fct_daily_post_statistics defined and how is it used?”</li><li>“What’s the path from raw LinkedIn exports to a specific metric on the dashboard?”</li></ul><p>So I pointed Graphify at the LinkedIn analytics repo and waited to see what it would learn.</p><p>That’s where the side quest began.</p><h3>II. What happened when Graphify met Databricks</h3><p>Running Graphify on the repo was immediately useful: it understood my Python modules, SQL files, and YAML configs, and built a graph that tied together jobs, pipelines, and resource definitions.</p><p>But two issues stood out very quickly.</p><h4>a. Notebooks were invisible</h4><p>Most of the critical pipeline logic in the LinkedIn project lives in Databricks notebooks: ingestion, cleaning, transformations from bronze to silver to gold, and the joins that power the final metrics.</p><p>Graphify, however, was blind to them.</p><p>Under the hood, Graphify uses a detect.py module to walk a directory and classify files by type: CODE, DOCUMENT, PAPER, IMAGE, and VIDEO, based on file extensions and some content heuristics.</p><p>When I looked at the extension sets in that file, here’s what I found:</p><ul><li>.py, .sql, .sh, .json, .tf, .hcl and many more were in CODE_EXTENSIONS.</li><li>.md, .txt, .html, .yml, .yaml were in DOC_EXTENSIONS.</li><li>.docx and .xlsx had a dedicated OFFICE_EXTENSIONS set and a special side‑car conversion path.</li><li>.ipynb was nowhere to be seen.</li></ul><p>That meant any .ipynb file would be classified as None and silently skipped by the detector. In practice, Graphify was constructing a rich graph of everything except the Databricks notebooks; unfortunately, these were exactly the files that described the pipeline.</p><h4>b. Dashboards were treated as generic JSON</h4><p>The open‑source LinkedIn repo includes a Databricks Lakeview dashboard exported as a .lvdash.json file. This JSON contains:</p><ul><li>Dataset definitions, with SQL queries referencing the gold tables.</li><li>Pages and widgets (charts, tables, filters) that make up the dashboard.</li><li>Ties back to the same tables used by notebooks and jobs.</li></ul><p>Graphify was treating this file as generic JSON code, because .json sits in CODE_EXTENSIONS.</p><p>That’s reasonable in general (a lot of config files are JSON) but it meant the dashboard structure wasn’t being interpreted as a <strong>semantic object</strong> in the graph (a dashboard with pages, widgets, and datasets). For my use case, that was a missed opportunity.</p><h3>III. Reading Graphify’s design before proposing anything</h3><p>Rather than jumping straight into code, I spent time reading Graphify’s own architecture:</p><ul><li>The README and docs, to understand the core goal: “turn any folder into a queryable knowledge graph for your AI assistant.”</li><li>The detect.py implementation, to see how file classification and side‑car conversion work, especially for Office files.</li><li>The tests under tests/test_detect.py and related modules, to see how aggressively the project guards against things like secret leakage and zip bombs.</li></ul><p>A few design principles stood out:</p><ul><li><strong>File classification is conservative.</strong> Unknown or risky file types are skipped rather than guessed. Sensitive directories and filenames are excluded via explicit patterns and generic keyword heuristics.</li><li><strong>Heavy formats get converted to markdown sidecars. </strong>PDFs, .docx, .xlsx, and Google Docs files are converted to markdown before being handed off to the LLM.</li><li><strong>Incremental updates are a first‑class concern.</strong> Graphify stores md5 hashes and mtimes in a manifest to know which files have changed between runs.</li></ul><p>Any change I proposed would need to respect those constraints rather than adding a one‑off hack for my specific project.</p><h3>IV. Designing notebook support the “Graphify way”</h3><p>I knew I wanted .ipynb files to be part of the graph, but <strong>not</strong> by pushing the raw JSON into the AST path. That would be noisy and brittle:</p><ul><li>The AST extractor would see the notebook as a JSON document, not as a sequence of code and markdown cells.</li><li>The word count would be inflated with metadata keys like &quot;cell_type&quot;, &quot;outputs&quot;, &quot;metadata&quot;.</li><li>Any future notebook schema changes would break assumptions.</li></ul><p>The solution was already in the codebase: <strong>copy what Graphify does for Office files.</strong></p><h4>a. Sidecar conversion: ipynb → markdown</h4><p>Graphify uses functions like docx_to_markdown and xlsx_to_markdown to convert office files into markdown sidecars in graphify-out/converted/, then indexes those sidecars as DOCUMENTs.</p><p>I proposed an analogous ipynb_to_markdown converter:</p><ul><li>Read the notebook JSON.</li><li>Iterate through cells in order.</li><li><strong>For markdown cells:</strong> emit the markdown as‑is.</li><li><strong>For code cells:</strong> wrap source in fenced code blocks (e.g. ```python).</li><li>Ignore outputs and most metadata.</li></ul><p>The result is a linear markdown representation of the notebook that preserves:</p><ul><li>The narrative structure (markdown cells).</li><li>The code that defines the pipeline (code cells).</li></ul><p>And crucially, it matches Graphify’s existing pattern: heavy or complex formats converted into a simpler, LLM‑friendly textual representation.</p><p>This is the conservative, universally-safe approach for the upstream PR — a single document representation any Graphify installation can handle. In graphifyy-skills (Section VII), I took this further by routing each cell type to a language-matched sidecar, so Python cells feed tree-sitter AST and SQL cells feed the SQL extractor directly.</p><h4>b. Separate extension set, not “just add .ipynb to code”</h4><p>In the pull request, I deliberately <strong>did not</strong> add .ipynb to CODE_EXTENSIONS. Instead, I introduced a dedicated NOTEBOOK_EXTENSIONS set and a routing branch in detect.py similar to OFFICE_EXTENSIONS.</p><p>The logic is:</p><ul><li>If a file has a notebook extension (.ipynb), convert it to markdown sidecar.</li><li>Treat the sidecar as a DOCUMENT for extraction and word counting.</li><li>Leave the original file untouched.</li></ul><p>This keeps concerns clean:</p><ul><li>CODE_EXTENSIONS remains the set of files that are directly parsed as code (e.g. .py, .sql, .ts).</li><li>OFFICE_EXTENSIONS and NOTEBOOK_EXTENSIONS are the “convert first” formats.</li><li>The incremental detection code doesn’t need to change; it works on sidecar paths just as it does today.</li></ul><h3>V. Filing the issue: explaining the gap (Issue #1497)</h3><p>With the design sketched out, I opened an issue on the Graphify repo: <strong>“feat: add Jupyter notebook (.ipynb) support via markdown sidecar extraction”</strong>.</p><p><a href="https://github.com/Graphify-Labs/graphify/issues/1497">feat: add Jupyter notebook (.ipynb) support via markdown sidecar extraction · Issue #1497 · Graphify-Labs/graphify</a></p><p>In the issue, I tried to do three things clearly:</p><h4><strong>a. Describe the current behaviour</strong></h4><ul><li>.ipynb is unrecognized by classify_file() and therefore skipped.</li><li>For a notebook‑heavy project like a Databricks pipeline, this leaves a large blind spot.</li></ul><h4><strong>b. Describe the expected behaviour</strong></h4><ul><li>.ipynb should be read and converted into a markdown sidecar.</li><li>The sidecar should be treated as a DOCUMENT so the LLM extraction path can “see” both code and markdown.</li></ul><h4><strong>c. Outline a concrete, minimal implementation</strong></h4><ul><li>A small converter function.</li><li>A new extension set and branch in detect.py.</li><li>Tests in test_detect.py to validate classification and conversion.</li></ul><p>The goal was to make it as easy as possible for maintainers and other contributors to assess whether this fit Graphify’s design and to propose improvements if needed.</p><h3>VI. Proposing the implementation (PR #1498)</h3><p>Issue #1497 framed the problem and proposed direction. PR #1498 implemented it.</p><p><a href="https://github.com/Graphify-Labs/graphify/pull/1498">Feat/ipynb notebook support by KunojiLym · Pull Request #1498 · Graphify-Labs/graphify</a></p><p>At a high level, the PR does the following:</p><ul><li>Adds NOTEBOOK_EXTENSIONS = {&quot;.ipynb&quot;} alongside the existing extension sets.</li><li>Introduces ipynb_to_markdown(path: Path) -&gt; str, mirroring the style and safety considerations of docx_to_markdown and xlsx_to_markdown.</li><li>Wires notebook conversion into detect() so sidecars are created in graphify-out/converted/ and queued for extraction as DOCUMENTs.</li><li>Adds tests to ensure that .ipynb files are no longer invisible; empty or malformed notebooks fail gracefully; and word counts come from the sidecar, not the JSON.</li></ul><p>At the time of writing, this PR is still under review. That’s normal for a popular project: maintainers are juggling many issues, and they have a responsibility to guard performance, security, and maintainability as usage grows.</p><p>Regardless of the final shape of the change, the process itself was instructive.</p><h3><strong>VII. The skills layer: a working solution today</strong></h3><p>While PR #1498 waits in the review queue, I published a parallel solution that works with stock Graphify right now: <a href="https://github.com/KunojiLym/graphifyy-skills">graphifyy-skills</a>.</p><p>The key design decision was to go further than a single markdown sidecar. Graphify already has purpose-built extractors for different file types: tree-sitter AST for Python, SQL AST for .sql, semantic extraction for markdown. Why collapse a notebook into one format when you can route each cell to the extractor it deserves?</p><p>ipynb-graphify does exactly that. It splits a Databricks notebook into per-language sidecars:</p><ul><li>%sql cells → .ipynb.sql : so table references are extracted by the SQL AST parser</li><li>Code cells → .ipynb.py : so functions, classes, and calls go through tree-sitter</li><li>%md and markdown cells → .ipynb.md : so narrative context feeds semantic extraction</li><li>%sh, %scala, %r cells : each routed to their own sidecar</li></ul><p>databricks-graphify adds a second converter, lvdash_split.py, which parses the Lakeview dashboard JSON, extracts dataset SQL queries with FROM/JOIN table references, and writes .lvdash.sql and .lvdash.md sidecars. The dashboard&#39;s data dependencies become first-class graph nodes, connected to the same gold tables that the notebooks write to.</p><p>No fork of Graphify is needed for this implementation; the sidecars use file extensions Graphify already understands.</p><p>The relationship between the skills repo and PR #1498 is complementary, not competing. The PR fixes <em>automatic detection</em>: notebooks are ingested without any user action, for everyone. The skills fix <em>comprehension quality</em>: you get richer, per-language graph nodes instead of a single flattened document. Both matter; they solve different problems at different layers.</p><h3>VIII. Why this side quest matters beyond one repo</h3><p>For my LinkedIn analytics project, the immediate benefit is obvious: if Graphify learns to understand .ipynb, the graph for that repo becomes much more complete.</p><p>But the reasons I think this is worth writing about go beyond one project and one PR.</p><h4>a. AI tooling is only as good as its domain fit</h4><p>Generic AI tools are powerful, but they often under‑serve domain‑specific practices:</p><ul><li>Databricks and data teams use notebooks heavily.</li><li>Analytics projects mix code, SQL, notebooks, YAML, dashboards, and Office exports.</li></ul><p>Out‑of‑the‑box tools can miss large parts of that picture. It’s on us, as users and practitioners, to notice these gaps and propose how to close them.</p><h4>b. Contributing beats complaining</h4><p>I could have stopped at “Graphify doesn’t understand notebooks, that’s a shame.” Instead, I chose to:</p><ul><li>Read the code and tests.</li><li>Designed solutions at two layers: a core detection fix (PR #1498) and a skill-layer extension (graphifyy-skills).</li><li>Published a working implementation immediately, without waiting for upstream review.</li><li>Opened an issue and PR to propose the core fix for the benefit of all users.</li></ul><p>Whether or not the maintainers accept the exact implementation, the conversation helps move the tool forward, and it sets an example for others who hit similar edge cases.</p><h4>c. Data engineers can (and should) shape AI tooling</h4><p>There’s a lot of talk about AI assistants helping engineers. The reverse is also true: engineers can help AI tooling better reflect how real projects are built.</p><p>For data and analytics practitioners, that means:</p><ul><li>Asking: “Does this tool understand notebooks? Dashboards? Orchestrators? Lakehouse schemas?”</li><li>When the answer is “not yet,” deciding whether to live with the gap or to co‑design the fix.</li></ul><p>This side quest was my small contribution to that second path.</p><h3>IX. If you want to try something similar</h3><p>If you’re working on notebook‑heavy analytics projects and want to improve how your AI assistant understands them, here’s a lightweight way to start:</p><ol><li><strong>Pick a real project.</strong><br>Choose a repo that matters to you, whether it is a Databricks pipeline, a Jupyter‑based experimentation repo, or an analytics codebase.</li><li><strong>Run Graphify (or a similar tool) on it.</strong><br>See which files show up in the graph and which don’t. Pay particular attention to notebooks, dashboards, and config artifacts.</li><li><strong>Inspect the tool’s detection logic.</strong><br>Look for extension lists and classification heuristics. Ask: <em>“What is this tool assuming about my project’s structure?”</em></li><li><strong>If you’re on Databricks or Jupyter: install </strong><a href="https://github.com/KunojiLym/graphifyy-skills"><strong>graphifyy-skills</strong></a><strong> and run /databricks-graphify or /ipynb-graphify on your repo.<br></strong> You’ll get per-language sidecars that Graphify can extract AST nodes from immediately — no fork, no PR needed.</li><li><strong>When you find a gap, write it down clearly.</strong><br>Capture current behaviour, expected behaviour, and a minimal proposed approach. This alone is valuable feedback to maintainers.</li><li><strong>If you have time, turn it into a PR.</strong><br>Respect the project’s patterns, add tests, and be open to feedback.</li></ol><p>And if you’re curious about the original LinkedIn analytics project that motivated this side quest, you can start with <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">Part 1</a> of the series and work your way through to the open‑source repo and dashboard.</p><p>Yingzhao Ouyang is an AI and data engineering specialist with a distinctive blend of humanities, business, and technical expertise, bringing a uniquely holistic perspective to enterprise data challenges that others with purely technical backgrounds miss. To find out more, visit his website at <a href="https://www.yzouyang.com/">https://www.yzouyang.com/</a> or follow his LinkedIn profile at <a href="https://www.linkedin.com/in/yzouyang/">https://www.linkedin.com/in/yzouyang/</a></p><p>This post is also available on Yingzhao Ouyang’s website: <a href="https://www.yzouyang.com/teaching-graphify-to-understand-databricks-notebooks/">https://www.yzouyang.com/teaching-graphify-to-understand-databricks-notebooks/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13856e2a09b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/teaching-graphify-to-understand-databricks-notebooks-13856e2a09b9">Teaching Graphify to Understand Databricks Notebooks</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AI Disruption Part 3: Designing for Judgment in an Age of Machines]]></title>
            <link>https://medium.com/@kunojilym/the-ai-disruption-part-3-designing-for-judgment-in-an-age-of-machines-981953529ce8?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/981953529ce8</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[public-policy]]></category>
            <category><![CDATA[ai-disruption]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 01:06:00 GMT</pubDate>
            <atom:updated>2026-06-09T01:06:00.647Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PRVKnGypteAgXa8s" /><figcaption>Human judgement still has a role to play. Photo by <a href="https://unsplash.com/@sasun1990?utm_source=medium&amp;utm_medium=referral">Sasun Bughdaryan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p><a href="https://medium.com/@kunojilym/the-ai-disruption-part-1-how-ai-is-reshaping-work-before-our-institutions-are-ready-713e2cd7c3a8">Part 1</a> diagnosed the fractures: an uneven AI dividend, eroding entry-level roles, and collapsing trust. <a href="https://medium.com/@kunojilym/the-ai-disruption-part-2-inside-the-sixth-technology-wave-754e75417b76">Part 2</a> mapped the structural fault underneath: a technology wave so compressed it fits inside a single career, targeting the apprenticeship ladder that turns raw execution into seasoned judgment.</p><p>Part 3 asks the practical question: What does a serious response require, or are we leaving the market to answer it by default?</p><h3>Who pays for the transition?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E4ayOAI2LUSzHD3f" /></figure><p><a href="https://fortune.com/2026/04/06/goldman-job-displacement-40-years-of-data-scarring-gen-z/">Goldman Sachs modelling</a> shows a stark pattern: workers displaced by technology accumulate ~10% less earnings growth over the following decade than their non-displaced peers. In 2026, someone is already paying for the AI transition.</p><p>Sometimes it looks like corporate reallocation — such as <a href="https://www.straitstimes.com/business/stanchart-to-slash-back-office-roles-to-boost-returns?ref=more-on-this-topic">Standard Chartered’s strategy</a> to replace “lower-value human capital” with automation. Sometimes it looks like structural hollowing.</p><p>A counter-trend is emerging: AI-linked layoffs are fueling <a href="https://www.forbes.com/sites/bernhardschroeder/2023/04/03/layoffs-are-fueling-a-new-wave-of-entrepreneurs-heres-how-you-can-join-them/">a wave of one‑person, AI‑augmented startups</a>. Equipped with agents and no-code tools, mid-career professionals are selling back the expertise their former employers no longer want on payroll. For some, this will be an upgrade — more autonomy and upside than a traditional job ever offered — but at the system level it still marks a shift: more of the transition risk is being carried on individual balance sheets.</p><p>This shift creates three distinct cost buckets:</p><ul><li><strong>Income and transition risk for exposed workers:</strong> <a href="https://www.brookings.edu/articles/how-ai-may-reshape-career-pathways-to-better-jobs/">Brookings estimates </a>6.1 million US workers combine high AI exposure with low adaptive capacity, meaning tight financial buffer and few transferable skills.</li><li><strong>Organisational restructuring:</strong> <a href="https://www.google.com/search?q=https://assets.kpmg.com/content/dam/kpmgsites/sg/pdf/2025/04/trust-attitudes-and-use-of-ai-global-report.pdf">BCG’s June 2026</a> survey of 12,000 workers found that firms pairing AI with end-to-end workflow redesign see a 25-point business impact, compared to just 5 points from tools alone. The dividend goes to those who redesign work, not those who merely buy software.</li><li><strong>Institutional infrastructure: </strong>Public training systems and safety nets still operate on a slower, legacy labour cycle. The cost of stagnation shows up as structural unemployment and eroding institutional legitimacy.</li></ul><p>Leaders do not avoid these costs by ignoring them; they simply shift them onto those with the least bargaining power.</p><h3>Three failure modes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q853gtJZfN9hWsXR" /><figcaption>From left to right: the productivity trap, the apprenticeship gap, and the trust-governance mismatch. Generated by Nano Banana 2.</figcaption></figure><p>Three structural failures already define this transition.</p><h4>The productivity trap</h4><p>Workers are saving time, but firms are failing to convert it into value. While Microsoft’s <a href="https://www.microsoft.com/en-us/worklab/work-trend-index">2026 Work Trend Index</a> confirms that culture dictates whether AI yields true leverage, many firms use AI merely to expand workloads or cut headcount without redesigning the underlying workflow. The result is frantic AI activity without increased human leverage.</p><h4>The apprenticeship gap</h4><p>When the intake layer of analysts, junior engineers, and associates thins, the damage outlasts the immediate cost savings. Entry-level roles are where professionals learn to interpret ambiguity, handle pressure, and weigh competing interests. Eroding this layer risks a future shortage of the higher-order judgment organisations claim they need.</p><h4>The trust-governance mismatch</h4><p>Trust in AI use is not rising with usage. <a href="https://survey.stackoverflow.co/2025/ai1-ai-tools-in-the-development-process">Stack Overflow’s 2025 developer survey</a> shows falling trust in AI outputs despite rising adoption. When compliance frameworks treat governance as a static legal hurdle, they miss the career risk felt by workers: being held accountable for automated outputs they cannot audit or challenge.</p><h3>How to respond?</h3><p>A systemic response requires action across three levels: individuals, organisations, and institutions.</p><h4>For individuals</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t0YNZSKRYtw55_ji" /><figcaption>Thinking about tasks. Photo by <a href="https://unsplash.com/@juliapotter?utm_source=medium&amp;utm_medium=referral">Julia Potter</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>The defining career question is simple: Which of your tasks are advanced pattern-matching, and which require interpretation under genuine uncertainty?</p><p>AI excels at the former. The <a href="https://www.weforum.org/reports/the-future-of-jobs-report-2025/">World Economic Forum’s Future of Jobs Report 2025</a> notes that analytical thinking is now the single most valued skill and that higher‑order cognitive and social skills are increasingly critical. Organisations do not pay for abstract “judgment”; they pay for people who can reliably make hard calls in messy situations — prioritising, de‑risking, and explaining those decisions to others. Those skills are often built in humanities programmes, but just as often in management, law, product, or years of front‑line responsibility.</p><p>The strongest profiles are hybrids:</p><ul><li><strong>Technically trained professionals</strong> must build capacities often underweighted in their training: long-form writing, corporate context, and ethics as a design constraint.</li><li><strong>Humanities-trained professionals</strong> must build the data fluency required to audit model outputs and collaborate with engineering teams.</li></ul><p>That said, relatively privileged knowledge workers are in the best position to act; in more precarious roles, institutions and organisations will need to do more of the heavy lifting to create real options.</p><h4>For organisations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IFYidjQDoS7-Bg0F" /><figcaption>Planning organisation transformation. Photo by <a href="https://unsplash.com/@kaleidico?utm_source=medium&amp;utm_medium=referral">Kaleidico</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>Three operational commitments separate real transformation from expensive experimentation:</p><ul><li><strong>Design the workflow, not just the stack:</strong> Restructure roles around human-agent collaboration and measure success by business outcomes, not tool adoption.</li><li><strong>Rebuild apprenticeship:</strong> Establish explicit guardrails for junior workflows. Junior staff should help design prompts, review outputs, and run post-mortems under senior supervision. It is less efficient today, but cheaper than discovering tomorrow that no one inside the organisation understands how the work gets done.</li><li><strong>Treat governance as an operating capability:</strong> Assign clear accountability for agent decisions and preserve a documented human-in-the-loop role for high-stakes outcomes.</li></ul><p>Cutting roles and renting an external bench of solo operators looks efficient on a quarterly balance sheet, but it leaves an organisation hollowed out and short of internal judgment.</p><h4>For institutions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VDlctIb30plJuThQ" /><figcaption>Singapore’s Parliament House against the CBD skyline. Photo by <a href="https://unsplash.com/@venusmajor?utm_source=medium&amp;utm_medium=referral">VENUS MAJOR</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p><a href="https://www.mom.gov.sg/newsroom/press-releases/2026/0430-formation-of-tripartite-jobs-council">Singapore’s latest measures</a> are a defensive play to buy time. On the surface, the <a href="https://www.mom.gov.sg/newsroom/press-releases/2026/0430-adoption-of-ai-among-firms">Ministry of Manpower’s April 2026 report</a> looks reassuring: resident unemployment held at 2.8% through 2025, and only 6.2% of AI-adopting firms cut headcount.</p><p><strong>But this data is a rearview mirror, not a crystal ball.</strong> It reflects a market operating on yesterday’s momentum. The true leading indicator is structural: 38.6% of Singaporean workers sit in high-exposure, low-complementarity roles. Recognizing that this buffer is expiring, policymakers are intervening now to build a safety net <em>before</em> the erosion of entry-level jobs turns into widespread layoffs.</p><p>Three design principles matter here:</p><ol><li><strong>Align training with labour-market intelligence:</strong> Systems must respond to real displacement risks, not just generic upskilling goals.</li><li><strong>Tie public subsidies to job redesign, not just model adoption:</strong> Funding AI use without a workforce transition plan simply accelerates the productivity trap.</li><li><strong>Build social partnership into transition decisions:</strong> Automation, retraining, and cost distribution are political-economy decisions, not just technical ones.</li></ol><p>Singapore is not a universal template; its high state capacity and tight tripartite system are unique. India’s IT services firms or Germany’s Mittelstand will face different constraints, for instance. <strong>Instead, it serves as a model of institutional urgency</strong>, showing that a serious response requires shaping the transition ahead of the market, rather than letting the headline numbers turn ugly.</p><blockquote>This essay focuses on design choices inside firms and public agencies, but it sits inside a wider contest over tax, labour law, and regulation that will ultimately decide how much of the AI dividend is shared and on what terms.</blockquote><h3>Acting before the numbers move</h3><p>The macro picture remains highly uncertain. <a href="https://www.weforum.org/reports/four-futures-for-jobs-in-the-new-economy-ai-and-talent-in-2030/">The WEF projects</a> 78 million net new jobs by 2030, but that figure masks massive churn: 92 million legacy roles displaced alongside 170 million new ones created.</p><p>The frontline evidence is far more urgent. While Anthropic’s <a href="https://www.anthropic.com/research/labor-market-impacts">observed-exposure study</a> does not show an immediate spike in aggregate unemployment, it reveals a telling metric: a 14% decline in job-finding rates for workers aged 22 to 25 in AI-exposed roles since the launch of ChatGPT.</p><p>This is a leading indicator. The cheapest time to respond to a structural shift is before it registers in headline unemployment statistics. The next five years are a narrow window where workflow redesign remains significantly cheaper than societal repair.</p><p>The sixth wave will generate immense economic value, but its distribution is an open political-economy question. If machines are handling the pattern-matching, the primary premium will not belong to prompt engineers. It will belong to people with the training and courage to say: <em>this pattern is wrong, this trade-off is unacceptable, this output does not match lived reality.</em> The ultimate shortage in an AI-saturated economy is judgment — and judgment must be intentionally built, not just deployed.</p><p><strong><em>Two questions to close with:</em></strong></p><ul><li><strong><em>For leaders:</em></strong><em> who is currently paying for your AI gains, and would they agree those terms are fair?</em></li><li><strong><em>For practitioners:</em></strong><em> what is one task you will automate this quarter, and what is one capability you will deliberately deepen that AI cannot easily replace?</em></li></ul><p><em>This is Part 3 of a series on the AI Disruption. </em><a href="https://medium.com/@kunojilym/the-ai-disruption-part-1-how-ai-is-reshaping-work-before-our-institutions-are-ready-713e2cd7c3a8"><em>Part 1</em></a><em> examined who is feeling the labour-market effects first and why trust is collapsing. </em><a href="https://medium.com/@kunojilym/the-ai-disruption-part-2-inside-the-sixth-technology-wave-754e75417b76"><em>Part 2</em></a><em> mapped the structural wave underneath: why this transition is compressed, why it targets the career ladder, and why institutions are still behaving as if they have a generation to respond.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=981953529ce8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AI Disruption Part 2: Inside the Sixth Technology Wave]]></title>
            <link>https://medium.com/@kunojilym/the-ai-disruption-part-2-inside-the-sixth-technology-wave-754e75417b76?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/754e75417b76</guid>
            <category><![CDATA[industrial-revolution]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[risk-management]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Wed, 03 Jun 2026 17:38:04 GMT</pubDate>
            <atom:updated>2026-06-03T17:38:04.639Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oHSTCOGD-Dt69EGX" /><figcaption>Where we are in the sixth technology wave. Generated by Nano Banana 2.</figcaption></figure><p><a href="https://medium.com/@kunojilym/the-ai-disruption-part-1-how-ai-is-reshaping-work-before-our-institutions-are-ready-713e2cd7c3a8">Part 1</a> showed how the AI dividend is already being allocated unevenly, with early-career white-collar workers absorbing much of the adjustment. Part 2 asks what kind of technological wave produces that pattern and why it is happening within a single career span.</p><h3>A shorter wave inside a longer history</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wDaWEfMAoaFXyHo4" /><figcaption>Five technology waves, with the sixth potentially within one compressed career span. Generated by Nano Banana 2.</figcaption></figure><p>Long before ChatGPT, economists like Nikolai Kondratiev and Joseph Schumpeter argued that capitalism evolves through long waves of clustered technological change rather than isolated inventions. From steam and electricity to software, each wave reorganized production, reshaped labor demand, and left institutions lagging behind technology.</p><p>Crucially, these technological waves are shortening. Standard reconstructions put the first modern industrial wave at roughly 60 years, later waves at 55, 50, 40, then about 30 years for the digital era. If AI marks the beginning of a sixth wave, the timescale may be 15 to 20 years.</p><p>How is this relevant? Previous industrial transitions were brutal because they unfolded across generations; those displaced at the start rarely lived to see the gains at the end. A compressed AI wave changes this calculus entirely: <strong><em>the shock, the backlash, the reskilling scramble, and the eventual stabilisation may all arrive within one working life.</em></strong></p><p>The length of the wave is only part of the story, however. These industrial transitions also delivered technological gains on one timetable while institutions, bargaining power, and labour-market adaptation moved on another. That same mismatch is now reappearing under far more compressed conditions.</p><h3>The Luddite analogy</h3><p><a href="https://www.history.com/articles/industrial-revolution-luddites-workers">The Original Luddites Raged Against the Machine of the Industrial Revolution | HISTORY</a></p><p>History labels the Luddites as mere enemies of progress, but the record tells a different story. They were skilled artisans watching the market value of their expertise collapse. Because their petitions targeted wages, standards, and transitional support rather than technology itself, the conflict was never truly about the machinery. It was about the terms on which a productivity transition would be imposed, and who would be asked to absorb the cost first.</p><p>For many textile workers, real earnings stagnated or fell for decades after mechanisation. <a href="https://fortune.com/2026/04/06/goldman-job-displacement-40-years-of-data-scarring-gen-z/">Recent work</a> by Goldman Sachs finds a similar scar today: workers displaced by technology accumulate roughly 10 percentage points less earnings growth over the following decade than comparable workers who are not displaced. Productivity jumps can be real while the path to those gains remains uneven and costly for specific groups.</p><blockquote>The Luddite analogy keeps resurfacing because it captures a durable political-economy problem. Every major technology wave promises long-run gains. The immediate question is who finances the transition in the meantime.</blockquote><h3>When the first bite lands on the ladder</h3><p><a href="https://medium.com/@kunojilym/the-ai-disruption-part-1-how-ai-is-reshaping-work-before-our-institutions-are-ready-713e2cd7c3a8">Part 1</a> showed that AI’s first clear impact has been on early-career hiring in exposed white-collar roles. Those roles are more than an employment category. They are the on-ramp into many professions and the mechanism by which organisations convert raw talent into mature judgment.</p><p>When the intake layer of analysts, junior engineers, associates, and coordinators thins out, the effects accumulate. Firms save on structured junior work, but they also compress the apprenticeship phase that used to produce future experts. Short-term efficiency gains can leave behind a shallower pool of people able to supervise, redesign, or challenge increasingly capable systems.</p><blockquote>This is why early labor-market data matters far beyond the entry-level cohort. The true risk is <strong>institutional amnesia</strong>: by automating away the structured apprenticeship layer, organizations are quietly dismantling their own talent-formation systems.</blockquote><h3>Why AI feels huge locally but small in aggregate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mB4dEiVdb-Lb8uJC" /><figcaption>Today’s AI is a pattern-matcher par excellente, but lacks the deeper understanding to intuit underlying theories. Generated by Nano Banana 2.</figcaption></figure><p>The gap between impressive demos and cautious macro forecasts is no contradiction; it reflects the difference between pattern‑matching and structural understanding.</p><p>A <a href="https://arxiv.org/abs/2602.06923">2026 paper on world models</a> illustrates this neatly: sequence models can predict trajectories with striking accuracy without recovering the underlying laws that generate them, acting like Kepler without Newton. Without carefully chosen inductive biases, these systems behave as advanced curve‑fitters rather than theorists.</p><p>Something similar is visible in today’s AI systems, which excel at <strong><em>descriptive work</em></strong>: recognising, transforming, summarising and ranking information. Those capabilities are commercially valuable, and they generate obvious local wins inside teams and workflows. <a href="https://knowledge.wharton.upenn.edu/article/how-ai-could-lift-productivity-and-gdp-growth/">Many economists</a> expect muted near-term macro gains because firms are merely optimizing individual tasks rather than redesigning whole organizations. While automating drafting, coding, or customer support yields immediate local returns, aggregate productivity will remain muted until firms move beyond task-level automation and begin redesigning how work is organised, decisions are made, and value is created.</p><blockquote>For leaders, that gap is the opportunity: the firms that go beyond task-level optimization and redesign workflows around AI will be the ones that show up in the macro data, not just in isolated case studies.</blockquote><h3>The irony at the heart of AI</h3><p>Modern AI stands on foundations that overlap awkwardly with human cognitive weaknesses. Turing’s theory of computation, Shannon’s information theory, von Neumann’s game theory and computer architecture, Holland’s genetic algorithms and the backpropagation work of Hinton and LeCun all turn messy reasoning problems into formal systems.</p><p>Many professional roles have been structured around executing these systems: drafting within fixed doctrines, running standard analyses, or writing code in familiar patterns. Once these structures become machine-executable, the role shifts. Value migrates upward toward deciding which problems to tackle, judging outputs, and recognizing when a model no longer fits reality.</p><p>That is one reason the labour effects are appearing so clearly in structured white-collar work. The old bargain in many professions was simple: do the formal work first, then earn the right to handle ambiguity later. If AI erodes the structured layer without replacing its apprenticeship function, <strong><em>organisations may end up with fewer people capable of higher-order judgment just as they need more of it</em></strong>.</p><h3>From sidecar to agent</h3><p>Rather than assisting with an isolated task, agentic systems plan multi-step execution, call tools, maintain intermediate states, and deliver finished work. This shift sounds incremental, but economically it marks a sharp break. While earlier automation left a residue of “glue work” that preserved human roles, agentic systems are beginning to absorb most of that residue.</p><p>If “sidecar” AI weakens the bottom rung of the career ladder, agentic AI threatens to bypass the middle entirely. The risk is no longer just losing isolated tasks; it is the automation of the coordinating, sequencing, and synthesis work that historically forged raw execution into seasoned mid-career judgment. Some glue work will survive as oversight and exception handling, but far less of it will be available as a training ground.</p><p>Consequently, the agentic transition will hit hardest in professions built around synthesis and workflow management. In <a href="https://arxiv.org/html/2604.00186v1">an agentic task exposure framework published in 2026</a>, legal, financial, and administrative roles in early-adoption cities show a rapid spike in exposure by 2027. The exact ranking matters less than the underlying mechanism: when AI owns the sequence rather than just accelerating the steps, it alters the shape of the job and chokes off the developmental path that produces future experts.</p><h3>Case Study: Singapore as an Early Warning System</h3><p><a href="https://www.mddi.gov.sg/newsroom/update-to-singapore-s-national-ai-strategy--refreshed-priorities-to-harness-ai-for-the-public-good-factsheet/">Update to Singapore&#39;s National AI Strategy: Refreshed Priorities to Harness AI for the Public Good (Factsheet)</a></p><p>Singapore offers a clear preview of the AI transition. The island-state combines exceptionally high occupational exposure with a labor market that is still only partially reorganized around it.</p><ul><li><strong>In most economies, disruption is hard to read:</strong> weak data, slow policy, and noisy signals obscure what is actually happening.</li><li><strong>Singapore’s signal is unusually clean:</strong> the economy is digitally intensive, policy capacity is high, and official data tracks exposure with enough precision to see who is at risk.</li></ul><p>So Singapore functions less as an exception and more as an early warning: if a country with serious workforce planning and high state capacity still faces a gap between AI capability and organizational readiness, less-prepared nations will face a much steeper challenge.</p><h3>Risk tools for an uncertainty problem</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tgjf-cq94czCaZXD" /><figcaption>Our probability-based risk frameworks are not accounting for the wider uncertainties concerning AI governance. Generated by Nano Banana 2.</figcaption></figure><p>The decline in trust is not a collective mood swing; it is an epistemic problem. Public intuition recognizes that this transition is poorly modelled.</p><p>Frank Knight’s classic <a href="https://www.econlib.org/library/Columns/y2018/Emmettriskuncertaintyprofit.html">distinction between risk and uncertainty</a> applies perfectly: <strong><em>risk</em></strong> assumes probabilities are known or estimable, while <strong><em>uncertainty</em></strong> begins when the model itself is in doubt. <a href="https://ideas.repec.org/a/eee/moneco/v69y2015icp97-113.html">Hansen and Sargent</a> take this a step further by distinguishing between <strong><em>ambiguity </em></strong>across several plausible models and <strong><em>misspecification: </em></strong>state where reality sits entirely outside the modeller’s assumptions.</p><blockquote>This is the world we are in today: <strong>institutions are trying to govern a moving target using frameworks built for a static world.</strong> This mismatch explains why trust is eroding even as usage climbs. Our current compliance models are too narrow to capture how deeply AI is rewriting incentives, bargaining power, and workplace boundaries.</blockquote><h3>What a serious pro-tech position should look like now</h3><p>The practical implication is not technological pessimism, but analytical discipline. The history of electrification and computing proves that massive productivity gains are real, but they demand complementary investment, systemic redesign, and time. The distributional lesson is equally stark: those gains are never shared smoothly unless institutions force the issue.</p><p>A serious pro-tech stance must reject two lazy narratives: the <strong><em>complacent optimism</em></strong> that aggregate growth will automatically rescue displaced workers, and the <strong><em>theatrical fatalism</em></strong> that treats technological progress itself as the enemy. Neither takes political economy seriously.</p><p>Here is the unvarnished truth:</p><blockquote>AI can <strong>generate immense economic value</strong> while simultaneously <strong>destabilizing career ladders</strong>, <strong>eroding corporate legitimacy</strong>, and <strong>exposing the limits of the models </strong>policymakers rely on to manage the transition.</blockquote><p>That is a much less comfortable story than either boosterism or panic, but it is the one most consistent with both economic history and the current evidence.</p><h3>The question that sits underneath all the others</h3><p>The defining question for the next decade is not whether new jobs will eventually appear (they likely will), but whether the transition can be made less punishing for the cohort living through it.</p><p>This is the reality of the sixth technology wave. It is not a story about faster software, but an economic shock deep enough to reshape production, labor, and institutional legitimacy on a timeline too short to ignore. The long run has arrived early.</p><p>While Part 1 diagnosed the first visible fractures, Part 2 maps the structural faults underneath. What Part 3 will have to reckon with is harder still: not just whether institutions can catch up, but whether they are willing to.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=754e75417b76" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AI Disruption Part 1: How AI is Reshaping Work Before Our Institutions are Ready]]></title>
            <link>https://medium.com/@kunojilym/the-ai-disruption-part-1-how-ai-is-reshaping-work-before-our-institutions-are-ready-713e2cd7c3a8?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/713e2cd7c3a8</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[disruption]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Wed, 27 May 2026 01:16:00 GMT</pubDate>
            <atom:updated>2026-05-27T01:16:00.529Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zovu9FI91eO5fox2" /></figure><p>AI is not just another productivity tool. It is changing who gets paid for productivity.</p><p>In this first part of the series, I look past the hype and examine what the data actually says about how AI is reshaping work, who is feeling it first, and why trust is collapsing even as adoption soars.</p><h3>The AI dividend isn’t evenly shared</h3><p>Most public debates about AI and jobs still start and end with “Will it replace us?”. That’s the wrong question.</p><p>The World Employment and Social Outlook by the International Labour Organization (ILO) shows that from 2004 to 2024, global workers’ <strong>output per hour rose by 58%</strong>, yet their <strong>share of income fell by 1.6%</strong> over the same period. That translates into a <strong>USD 2.4 trillion annual shortfall</strong> in labour income and a steadily shrinking share of the pie going to workers. The ILO explicitly warns that breakthroughs in generative AI could push workers’ share down further without deliberate policy intervention.</p><blockquote>For businesses and policymakers, that divergence is the macro backdrop AI is arriving into, not a distant future scenario.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bGSjyPsk04bOspdTH2GrRw.png" /><figcaption>Source: <a href="https://www.ilo.org/publications/flagship-reports/world-employment-and-social-outlook-trends-2025">https://www.ilo.org/publications/flagship-reports/world-employment-and-social-outlook-trends-2025</a></figcaption></figure><p>A separate IMF study goes further by modelling what happens when AI spreads through an advanced economy. It finds something that feels counterintuitive at first: wage inequality can actually fall slightly, while wealth inequality rises sharply. How so?</p><ul><li>AI heavily exposes high-income, white-collar roles; but these roles are also highly complementary to AI, so many of their incumbents see productivity gains rather than straight displacement.</li><li>Those same workers hold a disproportionate share of AI-appreciating assets (equities, corporate bonds, pension funds), so they benefit again as capital owners.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1FV0Zxx98Zfq2H05GrzPqQ.png" /><figcaption>Source: Rockall, E. J., Mendes Tavares, M., &amp; Pizzinelli, C. (2025). AI Adoption and Inequality. <em>IMF Working Papers</em>, <em>2025</em>(068). Retrieved May 26, 2026, from <a href="https://doi.org/10.5089/9798229006828.001">https://doi.org/10.5089/9798229006828.001</a></figcaption></figure><p>In other words, we’re not just talking about “jobs lost vs jobs gained”. We’re talking about who captures the AI dividend. The early evidence says capital and already-advantaged workers are winning that race.</p><h3>The canaries in the coal mine: entry-level white-collar</h3><p>If there is one labour market signal everyone should be paying attention to, it’s early-career employment.</p><p>Stanford’s Digital Economy Lab recently analysed high-frequency payroll data from the largest US payroll processor to see how AI is actually showing up in jobs. They found that:</p><ul><li>Early-career workers aged 22–25 in the most AI-exposed occupations have seen a <strong>16% relative decline in employment</strong> since the widespread adoption of generative AI.</li><li>More experienced workers in the <em>same</em> occupations saw employment remain stable or even grow.</li><li>The declines are concentrated exactly in roles where AI is more likely to automate tasks rather than augment them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*8D18gctOSp_ZfrDbsaQXJA.png" /><figcaption>Source: <a href="https://digitaleconomy.stanford.edu/publication/canaries-in-the-coal-mine-six-facts-about-the-recent-employment-effects-of-artificial-intelligence/">https://digitaleconomy.stanford.edu/publication/canaries-in-the-coal-mine-six-facts-about-the-recent-employment-effects-of-artificial-intelligence/</a></figcaption></figure><p>A separate study from Anthropic, using a new AI-exposure measure based on what their models can actually do today, finds a similar pattern at the hiring margin: job-finding rates for 22–25-year-olds in AI-exposed roles have dropped by about 14% since the release of ChatGPT. Interestingly, aggregate unemployment hasn’t spiked; many of these young workers simply never enter the statistics as “unemployed” in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ANkUZQou_8dkqyeP" /><figcaption>Source: <a href="https://www.anthropic.com/research/labor-market-impacts">https://www.anthropic.com/research/labor-market-impacts</a></figcaption></figure><p>And these are not routine factory jobs. The most AI-exposed occupations skew more educated, better paid, and more likely to be female or Asian.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*COgNaugzFL0NOMeE" /><figcaption>Source: <a href="https://www.anthropic.com/research/labor-market-impacts">https://www.anthropic.com/research/labor-market-impacts</a></figcaption></figure><p>In other words, AI is quietly attacking the on-ramp into white-collar careers, long before we see anything in headline unemployment numbers.</p><blockquote>If you’re a manager or policymaker, this is your future leadership bench — and they are the cohort seeing the quietest erosion of opportunity.</blockquote><h3>The APAC paradox: adoption leaders, fear leaders</h3><p>APAC is where this story becomes very real, very quickly.</p><p>BCG’s 2025 “AI at Work” survey of more than 4,500 employees shows that Asia-Pacific leads the world in workplace AI usage:</p><ul><li>78% of APAC employees use AI at work at least weekly, vs 72% globally.</li><li>70% of APAC frontline workers use generative AI regularly, vs just 51% globally.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*JNsH1jt-UrrHZf1XOLc6tA.png" /><figcaption>Source: <a href="https://www.bcg.com/publications/2025/ai-at-work-is-asia-pacific-leading-the-way">https://www.bcg.com/publications/2025/ai-at-work-is-asia-pacific-leading-the-way</a></figcaption></figure><p>But high adoption hasn’t translated into confidence. The same survey finds that workers in markets like Singapore, South Korea and Thailand report some of the highest fears about AI-related job loss in the region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSAibhmuD4dvYtUDhf3uQA.png" /><figcaption>Source: <a href="https://www.bcg.com/publications/2025/ai-at-work-is-asia-pacific-leading-the-way">https://www.bcg.com/publications/2025/ai-at-work-is-asia-pacific-leading-the-way</a></figcaption></figure><p>And the governance lag is clear. Only about 13% of respondents say AI agents are properly integrated into their workflows; most usage is informal, unstructured, and self-driven.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0fBYvGTAMmky_QCru6BbUg.png" /><figcaption>Source: <a href="https://www.bcg.com/publications/2025/ai-at-work-is-asia-pacific-leading-the-way">https://www.bcg.com/publications/2025/ai-at-work-is-asia-pacific-leading-the-way</a></figcaption></figure><p>Put simply: in APAC, AI isn’t a future scenario. It’s already on the frontline. But organisational design, skills, and governance have not caught up, and workers can feel it.</p><blockquote>For leaders outside APAC, this isn’t a niche regional story; it’s an early glimpse of what ‘normal’ might look like once AI saturates day-to-day work.</blockquote><h3>Singapore as a stress test</h3><p>Singapore is arguably the most interesting lab in the world for AI and labour.</p><p>The IMF’s 2024 deep dive into Singapore’s labour market estimates that <strong>around 77% of workers are in occupations with high AI exposure</strong>, compared with roughly 60% in other advanced economies. That’s no surprise; Singapore has intentionally built a high-skill, services-heavy economy.</p><p>The nuance is in the split:</p><ul><li><strong>38.9%</strong> of workers are in high-exposure, <strong>high-complementarity</strong> roles (managers, engineers, professionals) where AI is more likely to augment than replace.</li><li><strong>38.6%</strong> are in high-exposure, <strong>low-complementarity</strong> roles (clerical, admin, some ICT and sales roles) where AI can more easily automate core tasks.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9FSqWhmpsTYFCj6KXBA1yQ.png" /><figcaption>Source: Khan, S. A. (2024). Impact of AI on Singapore’s Labor Market — Singapore. <em>Selected Issues Papers</em>, <em>2024</em>(040), Article A001, A001. Retrieved May 26, 2026, from <a href="https://doi.org/10.5089/9798400285721.018.A001">https://doi.org/10.5089/9798400285721.018.A001</a></figcaption></figure><p>The structural risk is not evenly distributed:</p><ul><li>49% of female workers are in the high-exposure, low-complementarity bucket, vs 29% of male workers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*SgH92AkP41STs2vNiD6V6w.png" /><figcaption>Source: Khan, S. A. (2024). Impact of AI on Singapore’s Labor Market — Singapore. <em>Selected Issues Papers</em>, <em>2024</em>(040), Article A001, A001. Retrieved May 26, 2026, from <a href="https://doi.org/10.5089/9798400285721.018.A001">https://doi.org/10.5089/9798400285721.018.A001</a></figcaption></figure><ul><li>Half of workers aged 15–24 are in that same vulnerable segment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*VO9GlxMhulFCfVPP0VsZMw.png" /><figcaption>Source: Khan, S. A. (2024). Impact of AI on Singapore’s Labor Market — Singapore. <em>Selected Issues Papers</em>, <em>2024</em>(040), Article A001, A001. Retrieved May 26, 2026, from <a href="https://doi.org/10.5089/9798400285721.018.A001">https://doi.org/10.5089/9798400285721.018.A001</a></figcaption></figure><p>On the demand side, Singapore has already become a global outlier in AI job postings.</p><ul><li>The Stanford 2025 AI Index finds that <strong>3.27% of all job ads in Singapore mention AI</strong>, nearly double the US figure.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uan68RM17pZl9KvJ6YgZKA.png" /><figcaption>Source: <a href="https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf">https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf</a></figcaption></figure><ul><li>Indeed data from late 2025 shows that roughly <strong>one in six Singapore job postings referenced AI tools</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oRKb5GsL9tPorh4p" /><figcaption>Source: <a href="https://www.hiringlab.org/au/blog/2025/07/31/june-2025-sg-labour-market-update-postings-up-following-lengthy-decline/">https://www.hiringlab.org/au/blog/2025/07/31/june-2025-sg-labour-market-update-postings-up-following-lengthy-decline/</a></figcaption></figure><p>Yet there is an underlying paradox. In April 2026, Singapore’s Ministry of Manpower (MOM) released an AI adoption study that revealed the following:</p><ul><li>AI-skilled engineers and professionals see a <strong>13–25% increase in their salary</strong>.</li><li>AI mentions in Singapore job postings have surged, even with the cyclical decline in the number of postings by around 16% year-on-year.</li></ul><blockquote>So fewer roles overall; a larger share require AI skills just to get in the door; and those through that door enjoy a measurable wage bump.</blockquote><p>There’s another important layer. Surveying 2,560 firms employing about half a million workers, MOM found that:</p><ul><li><strong>71.5%</strong> of firms had not yet adopted AI.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*Z94C3c1T5NJdvKMCbRwu6Q.png" /><figcaption>Source: <a href="https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf">https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf</a></figcaption></figure><ul><li>Among adopters, only <strong>3.8%</strong> had integrated AI into core processes; most were still in planning or pilot stages.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*3aopjpJysqlL3jOuxia7jg.png" /><figcaption>Source: <a href="https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf">https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf</a></figcaption></figure><ul><li><strong>70.7%</strong> of AI-using firms reported higher worker productivity.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GBE23FDh7E_IvvAtUr6UoA.png" /><figcaption>Source: <a href="https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf">https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf</a></figcaption></figure><ul><li>Only <strong>6.2%</strong> reported headcount reductions attributable to AI; <strong>18.9%</strong> redesigned job functions, and <strong>13.9%</strong> created new AI-related roles.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMjSmjA-GkwOqg5dhoEVYA.png" /><figcaption>Source: <a href="https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf">https://stats.mom.gov.sg/iMAS_PdfLibrary/mrsd-LMAR-1Q-2026.pdf</a></figcaption></figure><p>The ministry’s conclusion is cautious but clear: at this stage, AI is “complementing rather than displacing labour”, but workers who do not upskill, especially in SMEs, risk being left behind.</p><blockquote>In other words, Singapore has bought time through strong institutions and proactive policy, but not immunity from structural pressure.</blockquote><h3>The trust and legitimacy gap</h3><p>All of this is happening against a backdrop of sharp, measurable collapse in trust.</p><p>In the developer community, AI adoption is up, but trust is down. Stack Overflow’s 2025 survey shows <strong>trust in AI output falling from 40% to 30%</strong> year-on-year, even as <strong>usage rises from 76% to 84%</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jjw4NN-ISzJ65V_zle-W7g.png" /><figcaption>Source: <a href="https://survey.stackoverflow.co/2024/ai#1-ai-tools-in-the-development-process">https://survey.stackoverflow.co/2024/ai#1-ai-tools-in-the-development-process</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-C8mZAJl2ZKNjQZqyQ8CA.png" /><figcaption>Source: <a href="https://survey.stackoverflow.co/2025/ai#1-ai-tools-in-the-development-process">https://survey.stackoverflow.co/2025/ai#1-ai-tools-in-the-development-process</a></figcaption></figure><p>There is a similar pattern among the broader public. A study by KPMG across 47 countries, including multiple APAC markets, found:</p><ul><li>On average, <strong>58%</strong> of respondents consider AI systems reliable, but only <strong>46%</strong> say they are willing to <em>trust</em> them.</li><li>Emerging economies (many in APAC) show <strong>higher trust</strong> than advanced economies: roughly <strong>3 in 5</strong> in emerging markets vs <strong>2 in 5</strong> in advanced economies.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kv3oQKjApCgkdYPGCVLQBA.png" /><figcaption>Source: <a href="https://assets.kpmg.com/content/dam/kpmgsites/sg/pdf/2025/04/trust-attitudes-and-use-of-ai-global-report.pdf">https://assets.kpmg.com/content/dam/kpmgsites/sg/pdf/2025/04/trust-attitudes-and-use-of-ai-global-report.pdf</a></figcaption></figure><p>In Singapore specifically:</p><ul><li><strong>50%</strong> of Singaporeans are willing to trust AI, with the rest unwilling or unsure.</li><li><strong>75%</strong> accept or approve of AI, but <strong>81%</strong> are concerned about negative outcomes.</li><li><strong>75%</strong> are unsure online content can be trusted because it may be AI-generated; <strong>89%</strong> want laws to combat AI-generated misinformation; and <strong>67%</strong> believe AI regulation is required.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VCB885eztJxFwEMryqqioQ.png" /><figcaption>Source: <a href="https://kpmg.com/content/dam/kpmgsites/xx/pdf/2025/05/trust-attitudes-and-use-of-ai-singapore-snapshot.pdf.coredownload.inline.pdf">https://kpmg.com/content/dam/kpmgsites/xx/pdf/2025/05/trust-attitudes-and-use-of-ai-singapore-snapshot.pdf.coredownload.inline.pdf</a></figcaption></figure><blockquote>That combination, of high perceived benefit, high perceived risk, and strong demand for regulation, is the textbook definition of a trust gap.</blockquote><p>And then there are the cultural signals. In May 2026, graduates at multiple US universities openly booed Eric Schmidt and other commencement speakers for optimistic AI remarks. That kind of visible pushback would have been almost unthinkable for an earlier generation of “tech optimism” narratives.</p><p><a href="https://www.theguardian.com/us-news/2026/may/18/eric-schmidt-ai-university-commencement-speech-booed">Ex-Google CEO Eric Schmidt booed after AI remarks at Arizona commencement</a></p><p>Taken together, these are not just vibes. They’re social signals that the adoption curve is running ahead of the trust curve, and that the cohort most exposed to AI’s labour market impacts is already voting with its voice.</p><h3>Where this series goes next</h3><p>Part 1 is deliberately diagnostic. The aim is not to sensationalise, but to ground the discussion in what we can actually see in the data:</p><ul><li>Productivity gains are real, but the AI dividend is not being shared evenly.</li><li>The first clear labour market impact is a quiet erosion of entry-level white-collar opportunities.</li><li>APAC and Singapore show what happens when adoption outpaces governance, and how strong institutions can buy time, but not immunity.</li><li>Trust is falling faster than many leaders realise.</li></ul><p>In Part 2, the series can move from diagnosis to trajectory: What plausible futures do these signals point to for APAC and Singapore? What happens if we treat the current signal as a Luddite moment; not in the cartoon version of “anti-tech”, but in the historical sense of skilled workers asking for fair terms in a structural transition?</p><p>If you had to pick one lens for Part 2 — <strong>global macro, APAC, or specifically Singapore</strong> — which would be most useful for you and your organisation? I’ll be using these signals to shape Part 2 of this series, so I’d love to hear your view.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=713e2cd7c3a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chips, Cells and Code: How Singapore is Applying its 40-Year Industrial Playbook to AI]]></title>
            <link>https://medium.com/@kunojilym/chips-cells-and-code-how-singapore-is-applying-its-40-year-industrial-playbook-to-ai-26889d5f5d91?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/26889d5f5d91</guid>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Thu, 19 Feb 2026 01:01:00 GMT</pubDate>
            <atom:updated>2026-02-19T01:01:00.921Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ID1RwAQFH0g4r-5gu7HbtQ.png" /><figcaption>Generated by Gemini 3 Pro via Perplexity</figcaption></figure><p>In the 1980s, the big bet was <strong>chips</strong>. Singapore set out to become a serious player in semiconductors and electronics, starting from low‑cost assembly and eventually becoming a key node in the global chip supply chain. In the 2000s, the spotlight shifted to <strong>cells</strong>. Biopolis and the broader biomedical sciences drive poured billions into labs, talent and cluster infrastructure to turn Singapore into “the Biopolis of Asia.”</p><p>In 2026, the new bet is <strong>code</strong> — specifically, Artificial Intelligence (AI).</p><p>Budget 2026 created a <a href="https://www.channelnewsasia.com/singapore/budget-2026-national-artificial-intelligence-council-ai-lawrence-wong-5925886">National AI Council</a> chaired by the Prime Minister, <a href="https://sbr.com.sg/economy/news/budget-2026-sg-launch-national-ai-missions-across-four-sectors">launched “AI missions”</a> in advanced manufacturing, connectivity, finance and healthcare, announced an expanded AI park at one‑north, committed over SGD 1 billion to a National AI R&amp;D Plan, and introduced <a href="https://www.straitstimes.com/singapore/budget-2026-companies-to-receive-tax-deductions-on-ai-spending-more-support">generous tax rebates and grants</a> for AI adoption.</p><p>If you work in data, AI or enterprise technology in Singapore, the real question is not whether AI is a hype cycle. It’s whether this third big bet will end up looking more like the chip story, the Biopolis story, or something in between.</p><h3>Act I — Chips: When the Bet Pays Off</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9j7Sup62EbO2uHWm" /><figcaption>Photo by <a href="https://unsplash.com/@hutters?utm_source=medium&amp;utm_medium=referral">Stan Hutter</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>How Singapore bet on semiconductors</h4><p>In the late 1960s and 1970s, the Economic Development Board (EDB) pushed hard to attract American and Japanese electronics manufacturers. Programmes like the Electronics Industry Promotion Scheme and new industrial estates helped bring in firms such as National Semiconductor, Fairchild, Texas Instruments and HP in the late 1960s and early 1970s.</p><p>The first phase was labour‑intensive electronics assembly. But by the 1980s and 1990s, the strategy shifted up the value chain:</p><ul><li>Wafer fabrication, through companies like Chartered Semiconductor (a joint venture involving Temasek and overseas partners).</li><li>Higher‑value components and precision engineering, supported by infrastructure in places like Jurong and supporting technical institutes.</li></ul><p>By the late 1980s, Singapore had become the world’s largest producer of disk drives and a significant player in semiconductors.</p><h4>What “success” looked like</h4><p>Fast‑forward to today and the numbers are striking:</p><ul><li>As of 2023, Singapore produces <a href="https://www.incorp.asia/blogs/singapore-semiconductor-industry-strategic-win-heres-why/"><strong>about 10% of the world’s semiconductor output</strong> and <strong>20% of global semiconductor equipment</strong></a>, and <strong>9 of the top 15</strong> global semiconductor companies operate here.</li><li>As of 2025, semiconductor manufacturing contributes roughly <a href="https://www.ame-st.com/industryinsight2"><strong>6% of Singapore’s GDP</strong></a> and supports more than 35,000 jobs.</li></ul><p>This is not a side story; <a href="https://techovedas.com/the-singapore-miracle-how-a-small-city-became-the-semiconductor-giant/">it is one of the core pillars of Singapore’s manufacturing base</a>.</p><h4>Hidden fragility and lessons</h4><p>The chip story, however, is not a straight line up:</p><ul><li>Chartered Semiconductor, once the world’s third‑largest contract chipmaker, struggled with losses in the 2000s and was <a href="https://www.networkworld.com/article/776543/data-center-atic-to-buy-chartered-fold-into-globalfoundries.html">acquired and merged into GlobalFoundries</a> in 2009 for USD 3.9 billion.</li><li>The once‑dominant disk drive cluster has largely disappeared as technologies and global competition shifted.</li></ul><p>Three practical lessons emerge for AI:</p><ul><li><strong>Compounding only happens with continuous upgrading.</strong> The chip bet paid off because Singapore kept climbing the value chain, from assembly to wafer fabs to advanced packaging and design.</li><li><strong>Over‑reliance on a few champions is risky.</strong> When a flagship player like Chartered Semiconductor falters, you feel it.</li><li><strong>Execution was the real advantage.</strong> Singapore’s strength lay less in inventing entirely new chip architectures and more in building the most efficient, reliable place to manufacture and move them.</li></ul><h3>Act II — Biopolis: When the Bet Is Only Half‑Won</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sRnLrItPXBj6dpHg" /><figcaption>Photo by <a href="https://unsplash.com/@_louisreed?utm_source=medium&amp;utm_medium=referral">Louis Reed</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>The biomedical big bet</h4><p>In 2000, Singapore launched the Biomedical Sciences Initiative. Biopolis opened in 2003 as a purpose‑built hub for biomedical research, co‑locating A*STAR institutes, public sector labs and private sector R&amp;D. The government invested billions in infrastructure and incentives to attract big pharma, biotech firms and top researchers.</p><p>Some observers at the time described this as an attempt to create a “fourth pillar” of the economy, alongside manufacturing, finance and logistics.</p><h4>Tangible successes</h4><p>On paper, as of 2023, <a href="https://www.channelnewsasia.com/today/big-read/singapore-biomedical-sector-progress-challenges-5141161">the outcomes look impressive</a>:</p><ul><li>Biomedical manufacturing output <a href="https://sg.finance.yahoo.com/news/ageing-population-rising-healthcare-needs-103502357.html">grew from around <strong>SGD 6 billion in 2000</strong> to about <strong>SGD 30 billion</strong> in the early 2010s</a>; in 2023, the biomedical cluster generated about <a href="https://www.jtc.gov.sg/about-jtc/news-and-stories/feature-stories/singapore-biomedical-ecosystem"><strong>SGD 38–40 billion in output</strong></a>.</li><li>In 2023, the sector accounted for roughly <strong>2.6% of GDP</strong> and employed over <strong>26,000</strong> people, with biomedical manufacturing job growth outpacing overall manufacturing.</li><li>As of 2023, there are <strong>60+ biopharma manufacturing facilities</strong> in Singapore and more than <strong>80 leading biomedical companies</strong> with significant operations here.</li><li>A growing startup scene has taken shape: fewer than 10 biotech startups in 2012 versus about 65 in 2023, with projections for further growth.</li></ul><p>Biopolis succeeded in making Singapore a recognised biomedical hub in Asia, with world‑class labs, manufacturing plants and a dense research community.</p><h4>Where it fell short</h4><p>Yet, 25 years on, even <a href="https://www.ciip.group.cam.ac.uk/reports-and-articles/singapores-biomedical-cluster-lessons-from-two-decades-of-innovation-and-m">sympathetic analyses</a> acknowledge that “the big league remains elusive.”</p><ul><li>There are still relatively few blockbuster drugs originating from Singapore compared to the scale of investment.</li><li>A CNA long‑form article on the sector notes that Singapore is strong in discovery and manufacturing, but weaker in taking IP through late‑stage clinical development into large global products.</li><li>Graduate outcomes tell a story too: in 2023, life sciences graduates had lower median starting salaries and lower full‑time employment rates than the overall graduate average, suggesting an oversupply of generic talent relative to high‑value roles.</li><li>Philip Yeo, the former EDB chairman who drove the early push, has argued that scientists need to be more entrepreneurial and that the ecosystem needs more sophisticated, long‑horizon capital for translation.</li></ul><p>In short: the Biopolis bet clearly improved output, jobs and capabilities, but has not yet produced a proportionate number of globally dominant biomedical companies or products.</p><h4>Lessons for AI</h4><p>Biopolis offers some cautionary signals for AI:</p><ul><li><strong>Infrastructure ≠ impact.</strong> World‑class buildings, equipment and research output do not automatically translate into commercial outcomes.</li><li><strong>Talent must match demand.</strong> Training lots of people in a hot discipline is not enough if the downstream roles and companies don’t exist at scale.</li><li><strong>Commercialisation paths matter.</strong> Without entrepreneurial, market‑driven translation, a sector can stall at “good jobs and publications” rather than creating globally defining firms or platforms.</li></ul><h3>Act III — AI: Singapore’s Third Big Bet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s0RPqbsRKVRS8eUk" /><figcaption>Photo by <a href="https://unsplash.com/@steve_j?utm_source=medium&amp;utm_medium=referral">Steve Johnson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Budget 2026’s AI agenda fits squarely into this lineage.</p><h4>What Budget 2026 actually does</h4><p>Key elements of the AI push include:</p><ul><li><strong>National AI Council</strong>: A PM‑chaired council to set AI strategy, coordinate agencies and align regulation, talent and infrastructure.</li><li><strong>AI missions in four sectors</strong>: Advanced manufacturing, connectivity/logistics, finance and healthcare — areas where Singapore already has strong clusters.</li><li><strong>AI park at one‑north + NAIRD Plan</strong>: Expansion of the AI cluster at one‑north (the “Lorong AI” area) and a <strong>&gt;SGD 1 billion National AI R&amp;D Plan</strong> for 2025–2030 focused on applied and foundational AI.</li></ul><p>And if we zoom in on enterprise incentives, we find:</p><ul><li>Enhanced <strong>Enterprise Innovation Scheme</strong> with a <strong>400% tax deduction</strong> on up to SGD 50,000 of qualifying AI expenditure in YA2027 and YA2028.</li><li>A <strong>Champions of AI</strong> programme to support end‑to‑end AI transformation (technology, process and workforce).</li><li>An expanded <strong>Productivity Solutions Grant</strong> (PSG) list to cover more AI and AI‑enabled tools, especially for SMEs.</li><li><strong>Talent measures</strong>: Six months’ free access to premium AI tools for Singaporeans who enroll in selected AI‑related courses, plus redesigned AI learning pathways under SkillsFuture.</li></ul><p>A ChannelNewsAsia commentary described this as a “shift from aspiration to execution” in AI policy.</p><p><a href="https://www.channelnewsasia.com/commentary/budget-2026-ai-council-missions-singapore-5926811">Commentary: When it comes to AI, Budget 2026 marks a shift from aspiration to execution</a></p><h4>How it rhymes with chips and Biopolis</h4><p>The AI agenda clearly echoes both earlier bets:</p><ul><li>Like <strong>chips</strong>, it focuses on sectors where Singapore already has comparative strengths: manufacturing, logistics, finance, healthcare. The goal is not to out‑research frontier labs, but to become the place where AI is deployed <strong>reliably and at scale</strong> in real industries.</li><li>Like <strong>Biopolis</strong>, it leans heavily on <strong>clusters and R&amp;D</strong>: AI park, NAIRD funding, national missions coordinated at the centre.</li></ul><p>The big question is which pattern will dominate.</p><h4>What history suggests could go right</h4><p>If the AI push follows the <strong>chips</strong> trajectory:</p><ul><li>AI becomes deeply embedded in the operations of factories, ports, airlines, banks and hospitals, which will lift productivity in ways that persist for decades.</li><li>Singapore becomes a <strong>trusted deployment hub</strong> for AI in regulated sectors, much as it is for financial services and advanced manufacturing today.</li><li>Value‑chain upgrading continues: from basic chatbot pilots to AI‑native services, integrated data platforms, and exportable solutions.</li></ul><h4>What history suggests could go wrong</h4><p>If it repeats the <strong>Biopolis</strong> pattern:</p><ul><li>AI park and NAIRD produce high‑quality research and many pilots, but <strong>relatively few production‑grade systems at scale</strong> in the mission sectors.</li><li><strong>Training capacity overshoots:</strong> thousands of people get “AI‑flavoured” training, but the number of genuinely applied roles in logistics, manufacturing, finance and healthcare grows more slowly.</li><li><strong>Enterprise AI devolves into “AI theatre”:</strong> proofs‑of‑concept that look good in press releases but never make a dent in core operations or P&amp;L.</li><li>Over‑reliance on a handful of large global vendors leads to <strong>fragile dependencies</strong>, echoing the risks of clustering around a few champions in chips or biomed.</li></ul><p>The Budget 2026 measures are a strong start. But history suggests that execution — by enterprises and practitioners, not just the state — will determine whether this becomes another semiconductor‑level success or a Biopolis‑style half‑victory.</p><h3>What This Means for Enterprises</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ZYRgbDt7YB7ZFF4GsAO5bA.png" /><figcaption>Generated by Perplexity</figcaption></figure><p>For senior leaders in Singapore‑based enterprises, especially in finance, logistics, manufacturing and healthcare, three implications stand out.</p><h4>1. Design AI programmes that look more like chips than Biopolis</h4><ul><li><strong>Align to national missions.</strong> AI roadmaps should explicitly reference one or more of the four mission sectors. For a bank or insurer, that means focusing on AI for underwriting, claims, fraud, capital and compliance — not just generic copilots.</li><li><strong>Take advantage of the Budget incentives.</strong> Structure projects so parts of the spend qualify for: a) EIS 400% AI deductions (up to SGD 50,000 per YA 2027–2028); b) PSG support for off‑the‑shelf tools; and c) Champions of AI transformation support where appropriate.</li><li><strong>Invest in AI‑ready foundations.</strong> Semiconductors paid off because Singapore kept upgrading its factories and skills. For AI, that means robust data platforms, governance, MLOps and change management, not just buying tools.</li></ul><h4>2. Avoid Biopolis‑style gaps</h4><ul><li><strong>Bake commercialisation in from day one.</strong> Treat AI initiatives as products with owners, metrics and adoption plans, not as research experiments.</li><li><strong>Guard against talent mismatch.</strong> Be clear about the roles needed (e.g. data engineers, MLOps, applied ML in specific domains) and avoid the temptation to hire generalist “AI” talent without a roadmap for how they will be used.</li><li><strong>Measure real impact.</strong> Use the same discipline you’d apply to a core system migration: track productivity, error rates, throughput, risk outcomes — not just “number of pilots.”</li></ul><h3>What This Means for Practitioners</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*pjGJINiOHtCiuvGZH9LyVg.png" /><figcaption>Generated by Perplexity</figcaption></figure><p>For data and AI practitioners — and those trying to transition into the field — Singapore’s track record suggests a few concrete moves.</p><h4>1. Pick a mission, not just a model</h4><p>Singapore’s past bets have always been <strong>sector‑anchored</strong>: chips, biomed, finance, logistics. The same will be true for AI.</p><p>Rather than branding yourself as “an AI person,” it will be more powerful in Singapore’s context to become, for example:</p><ul><li>An AI practitioner in banking/insurance,</li><li>A data/AI engineer in advanced manufacturing,</li><li>A healthcare AI specialist, or</li><li>A logistics/supply chain AI specialist.</li></ul><p>That is where most of the Budget 2026 energy — and thus the most durable roles — are likely to be.</p><h4>2. Aim for deployment, not just exposure</h4><p>The signal that matters is not “I’ve taken AI courses,” but “I have deployed AI into a real workflow in a mission sector.”</p><p>Budget 2026’s offer of six months’ free access to premium AI tools with selected courses is a clear nudge: your learning journey is expected to include <strong>hands‑on work</strong> with real tools and data, not just slideware. Use that to build a portfolio of concrete projects — especially ones that map neatly to manufacturing, finance, logistics or healthcare use cases.</p><h3>Closing: The Spectrum Between Chips and Biopolis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*YSbmzjeEpEjLyWju88CEkQ.png" /><figcaption>Generated by Perplexity</figcaption></figure><p>Over the last 40 years, Singapore has shown it can change its economic destiny by making a handful of big, disciplined bets.</p><ul><li>The semiconductor bet shows what happens when that bet compounds for three decades: a major share of global output, deep local capability, and a durable economic pillar.</li><li>The Biopolis bet shows what happens when world‑class research and infrastructure are not yet matched by an equally strong commercial engine: good jobs and capabilities, but still short of the original ambition.</li></ul><p>Budget 2026’s AI agenda is the <strong>third big bet</strong> in this sequence. It is not a passing initiative; it is the start of a multi‑decade structural shift.</p><p>The open question is where AI will land on the chips–Biopolis spectrum, and whether enterprises and practitioners treat this Budget as a prompt for serious, long‑term transformation, or just another opportunity for AI‑themed slides.</p><p>Yingzhao Ouyang is an AI and data engineering specialist with a distinctive blend of humanities, business, and technical expertise, bringing a uniquely holistic perspective to enterprise data challenges that others with purely technical backgrounds miss. To find out more, visit his website at <a href="https://www.yzouyang.com/">https://www.yzouyang.com/</a> or follow his LinkedIn profile at <a href="https://www.linkedin.com/in/yzouyang/">https://www.linkedin.com/in/yzouyang/</a></p><p>This post is also available on Yingzhao Ouyang’s website: <a href="https://www.yzouyang.com/chips-cells-and-code-how-singapore-is-applying-its-40-year-industrial-playbook-to-ai/">https://www.yzouyang.com/chips-cells-and-code-how-singapore-is-applying-its-40-year-industrial-playbook-to-ai/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26889d5f5d91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own LinkedIn Analytics Part 12: What’s Next? Open Sourcing and Community]]></title>
            <link>https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-12-whats-next-open-sourcing-and-community-edd57f3cea20?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/edd57f3cea20</guid>
            <category><![CDATA[data-engineering-101]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 17:50:00 GMT</pubDate>
            <atom:updated>2026-01-19T17:51:29.359Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*yXaHnZC0RQYQL46I58Fj5g.png" /></figure><p>This is the final part of my blog series on building a LinkedIn analytics pipeline, from <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">ingestion</a> of Excel exports all the way to a production-style <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">dashboard</a>, complete with <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">orchestration</a>, <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">maintainability</a> and <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd">observability</a>. In the previous post, I went over the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned-4d08f03ae1a8">key takeaways and lessons learned</a> when building this data product. Now it’s time to look to the future.</p><h3>Article Series</h3><ol><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">Why It Matters (and What You’ll Learn)</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">Choosing the Data Source</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-3-creating-the-data-architecture-287ef0a81f85">Creating the Data Architecture</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">Ingesting Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-5-cleaning-and-transforming-data-7478352d73c8">Cleaning and Transforming Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-6-modelling-the-data-9ea2daa78bcf">Modelling the Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">Dashboard Design for Insights and Impact</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">Orchestrating and Automating the Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">Making a Maintainable Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd">Observing the Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned-4d08f03ae1a8">Key Takeaways and Lessons Learned</a></li><li><strong><em>You are here </em></strong>➡️What’s Next? Open Sourcing and Community</li></ol><h3>TL;DR</h3><ul><li>The full Databricks‑based LinkedIn analytics pipeline is now <strong>open‑sourced</strong>, so you can clone it, plug in your own exports, and start analysing your content end‑to‑end.</li><li><strong>Future work</strong> spans richer data (articles, multimedia, post features), deeper analytics (DS/ML), and automation (GenAI assistants, CI/CD with GitHub Actions).</li><li>This installment is an <strong>invitation</strong>: fork the project, adapt it to your context, and share back improvements or findings with the community.</li></ul><h3>I. Open Source Repository</h3><p>First, here’s the link to the repository that you have all been waiting for.</p><p><a href="https://github.com/KunojiLym/databricks-linkedin-analytics">GitHub - KunojiLym/databricks-linkedin-analytics</a></p><p>This is where you’ll find Databricks notebooks, table and pipeline configurations, and dashboard definitions that mirror what has been demonstrated and shared throughout the series. At the same time, the repository is meant to complement this blog series while still being able to stand on its own. The idea is that anyone can clone or fork this codebase to experiment with their own LinkedIn analytics, or adapt the codebase to create a different data product.</p><p>If there are any issues that you encounter, or if you have any ideas for improvement, feel free to contribute to or comment on the repository. See <a href="https://github.com/KunojiLym/databricks-linkedin-analytics/blob/main/databricks_linkedin_analytics/CONTRIBUTING.md">CONTRIBUTING.md</a> for more details</p><h3>II. Future Plans</h3><p>There is a lot of work that can still be done with this project, some of which could warrant its own blog series. The following list is not comprehensive and is in no particular order of importance or urgency; some of this was touched on in the previous post reflecting on <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned-4d08f03ae1a8">lessons learned</a>.</p><ol><li><strong>Enhancements to data:</strong> There are post features that I did not ingest such as post type or holidays; I also did not ingest LinkedIn articles or any form of associated multimedia.</li><li><strong>Enhancements to ingestion:</strong> The current method of uploading files directly in the Databricks UI is not ideal; there are ways to ease the pain, including building an app on Databricks to simplify the upload process.</li><li><strong>Enhancements to dashboard:</strong> The current dashboard has plenty of room for improvement, especially as I discover additional requirements for my own analysis.</li><li><strong>Enhancements to documentation:</strong> Since I intend the repository to be able to stand on its own, eventually I will want to include a full tutorial/walkthrough in the repository documentation.</li><li><strong>Data science and machine learning:</strong> The amount of data that I collect will eventually increase to the point where it makes sense to perform deeper exploratory analytics on the data. Machine learning techniques can then be leveraged, such as classification to determine clusters of topics and predictive analytics to forecast how well different types of posts might fare or to identify the best timings for posts.</li><li><strong>CI/CD with GitHub Actions:</strong> I alluded to this when discussing <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">improvements to maintainability</a>, but the CI/CD of this data product is at present triggered via the Databricks UI and could be further automated by GitHub Actions.</li><li><strong>Generative AI:</strong> These days, dashboards are not the only way to derive insights from the data. Building a chatbot with Databricks Genie can prove to be a fruitful endeavour. There are other potential uses for GenAI, especially when one considers the potential of using an agentic framework to suggest topics for exploration.</li></ol><h3>III. Final Thoughts</h3><p>Over the past twelve parts, this project has grown from a simple personal experiment into a reusable reference stack for building creator‑grade analytics on Databricks. Rather than declaring it “done,” open‑sourcing the code is a way to keep it alive in the hands of other practitioners who see different problems and opportunities than one person ever could.</p><p>If you adapt this pipeline, whether for your own LinkedIn, for a team’s content program, or as a teaching tool, I would love to hear what you discover and what you change.</p><blockquote>What would you build next on top of this: richer data, ML‑driven insights, or GenAI‑powered assistants?</blockquote><p>Yingzhao Ouyang is an AI and data engineering specialist with a distinctive blend of humanities, business, and technical expertise, bringing a uniquely holistic perspective to enterprise data challenges that others with purely technical backgrounds miss. To find out more, visit his website at <a href="https://www.yzouyang.com/">https://www.yzouyang.com/</a> or follow his LinkedIn profile at <a href="https://www.linkedin.com/in/yzouyang/">https://www.linkedin.com/in/yzouyang/</a></p><p>This post is also available on Yingzhao Ouyang’s website: <a href="https://www.yzouyang.com/build-your-own-linkedin-analytics-part-12-whats-next-open-sourcing-and-community/">https://www.yzouyang.com/build-your-own-linkedin-analytics-part-12-whats-next-open-sourcing-and-community/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edd57f3cea20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own LinkedIn Analytics Part 11: Key Takeaways and Lessons Learned]]></title>
            <link>https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned-4d08f03ae1a8?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d08f03ae1a8</guid>
            <category><![CDATA[retrospectives]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering-101]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 01:02:29 GMT</pubDate>
            <atom:updated>2026-01-19T18:57:16.078Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-knEiqMouU2vHtwnsp-JIA.png" /></figure><p>If you’ve followed along and made it this far in the blog series, congratulations. You have a working LinkedIn analytics pipeline: from <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">ingestion</a> of Excel exports all the way to a production-style <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">dashboard</a>, complete with <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">orchestration</a>, <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">maintainability</a> and <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd">observability</a>.</p><p>Now it’s time to recap and review.</p><h3>Article Series</h3><ol><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">Why It Matters (and What You’ll Learn)</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">Choosing the Data Source</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-3-creating-the-data-architecture-287ef0a81f85">Creating the Data Architecture</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">Ingesting Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-5-cleaning-and-transforming-data-7478352d73c8">Cleaning and Transforming Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-6-modelling-the-data-9ea2daa78bcf">Modelling the Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">Dashboard Design for Insights and Impact</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">Orchestrating and Automating the Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">Making a Maintainable Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd">Observing the Pipeline</a></li><li><strong><em>You are here </em></strong>➡️Key Takeaways and Lessons Learned</li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-12-whats-next-open-sourcing-and-community-edd57f3cea20">What’s Next? Open Sourcing and Community</a></li></ol><h3>TL;DR</h3><ul><li>You now have <strong>an end‑to‑end LinkedIn analytics pipeline</strong> that ingests Excel exports into a medallion‑style architecture and surfaces insights on a Databricks dashboard.</li><li>The project doubles as a <strong>production‑like reference implementation</strong>, showcasing practices such as incremental ingestion, idempotent tasks, version control, and data quality and observability patterns.</li><li><strong>Databricks Free Edition</strong> proved fit for a solo, learning‑oriented stack, but also surfaced real constraints (manual exports, Free‑tier limits) that highlight how an enterprise deployment would differ.</li><li>There is rich headroom for <strong>future enhancements</strong>, including deeper analytics, ML models, GenAI‑powered interactions with the data, and ingestion of multimedia and article content.</li></ul><h3>I. Recap: What Did We Build?</h3><h4>a. The context</h4><p>In the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">opening post</a> of the series, I outlined why I wanted to build my own LinkedIn analytics data product instead of relying on existing tools. To sum up:</p><ul><li>Native LinkedIn analytics were too limited and clunky for the type of analysis I wanted to do.</li><li>Third‑party analytics tools were powerful but expensive overkill for a single‑creator workflow.</li><li>Databricks Free Edition offered a realistic way to prototype a small, end‑to‑end production‑style data product.</li><li>The project would double as a reference implementation that budding data engineers could learn from, not just a personal dashboard.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wicLidTTa66_eUnb26d-rw.png" /><figcaption>The chosen method for ingesting LinkedIn data from the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">second post</a>.</figcaption></figure><p>I explored my options for <strong>data sources</strong> in the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">second post</a>, and eventually chose manual Excel exports as “the least bad” approach for an individual creator. The <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-3-creating-the-data-architecture-287ef0a81f85">third post</a> built on that decision by articulating a <strong>medallion-style data architecture</strong> that would serve as the scaffold for everything else in the series.</p><h4>b. From raw exports to insights</h4><p>The next few posts focused on bringing the raw data in the exported LinkedIn Excel files through a medallion-style data journey all the way to insights on a dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ratJi3_QR6VcWtoZWRISNA.png" /><figcaption>The data flow for the LinkedIn analytics data product (summary graphic of complete data architecture from the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">eighth post</a>).</figcaption></figure><ul><li>The <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">fourth post</a> detailed the process of <strong>ingesting</strong> the raw Excel files into the bronze layer, with minimal transformations. I also augmented the Excel files with additional data from crawling the LinkedIn post links embedded in those files as well as manual patching.</li><li>The <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-5-cleaning-and-transforming-data-7478352d73c8">fifth post</a> expounded on the <strong>transformation</strong> of bronze data into silver-layer tables that are cleaned and standardised.</li><li>In the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-6-modelling-the-data-9ea2daa78bcf">sixth post</a>, I <strong>modelled</strong> the silver data into fact and dimension tables (or materialized views) on the gold layer for my analytic needs.</li><li>Then in the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">seventh post</a>, I built a <strong>dashboard</strong> on top of those gold-layer tables that finally provided me with LinkedIn insights.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*bqgnFZVAXAJcEjYz.png" /><figcaption>A dashboard panel screencap showing LinkedIn insights from the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">seventh post</a>.</figcaption></figure><p>This is where most guides to building personal projects stop, but for the benefit of budding data engineers, I wanted to go further than just a dashboard.</p><h4>c. Making it production-ready</h4><p>Once the proof-of-concept was in place, it was time to make the pipeline production-ready. What did that mean in practice?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ku_ly-OdZGTBUmeHIBxX_w.png" /><figcaption>The complete setup for the LinkedIn analytics pipeline from the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">eighth post</a>.</figcaption></figure><ul><li>I <strong>orchestrated and automated</strong> the pipeline by adding file arrival triggers and linking the individual pieces of code into Databricks Jobs and Pipelines, as detailed in the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">eighth post</a> in the series.</li><li>Then I made the pipeline <strong>maintainable</strong> in the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">ninth post</a> by creating a Databricks Asset Bundle within a Git repository that ensured that everything (or as much as possible) was version controlled and documented.</li><li>Finally, in the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd">tenth post</a>, I outlined the steps taken to improve the <strong>observability</strong> of the pipeline so that maintainers can be alerted to any issues and triage them in a timely manner.</li></ul><h3>II. What Went Well?</h3><h4>a. Personal learnings from the project</h4><ul><li>I set out to explore <strong>Databricks Free Edition</strong> as a way to build an end-to-end, production-like stack, and for the most part, it proved a good fit for a solo, learning-oriented project.</li><li>Though I use Databricks regularly in my day job, this was still my first time interacting with some of its features such as Dashboards and Databricks Asset Bundles, and it was a great opportunity to <strong>deepen my experience</strong> with this major data platform.</li></ul><h4>b. Teachable moments in the blog series</h4><p>Throughout the series, I highlighted <strong>best practices</strong> in production and enterprise environments. My aim was to show how these ideas appear in real code and architecture diagrams rather than in abstract checklists.</p><ul><li>I established a clear <strong>separation of concerns </strong>with my medallion architecture (bronze ingestion and patching; single source of truth in the silver layer; gold semantic modelling).</li><li>I repeatedly <strong>called out where the constraints of Databricks Free Edition and manual Excel exports differ from an enterprise setup</strong>, so readers could see how to adapt the same patterns to a more robust environment.</li><li>I used <strong>real‑world patterns</strong>, such as fan‑in orchestration, materialized views for gold tables and idempotent pipeline design, to connect everyday LinkedIn analytics to the kinds of workflows data engineers encounter in production.</li></ul><h3>III. What Can Be Done Better?</h3><h4>a. Technical limitations</h4><ul><li>Because I am an individual user rather than a business, I could only do <strong>manual LinkedIn Excel exports and uploads</strong>. There is scope here for a business process automation or even an agentic workflow to remove this manual element, but it needs to be explored with care, especially around LinkedIn’s terms of service and rate limits.</li><li>Databricks Free Edition has <strong>constraints</strong> such as table limits, a single region and workspace, and limited serverless compute, which meant that I couldn’t fully demonstrate what an enterprise-level deployment would look like. If there is sufficient interest, I may explore that in a follow-up series using an Enterprise workspace instead of the Free Edition.</li><li><strong>Some Databricks capabilities are still evolving</strong> beyond Free Edition constraints; Dashboards, the IDENTIFIER clause and Databricks Asset Bundles are prime examples, with feature gaps that Databricks is actively closing over time.</li></ul><h4>b. Potential technical improvements</h4><ul><li>A lot more can be done on the analytics front, especially as new data comes in, from further <strong>exploration of the data</strong> to <strong>machine learning models</strong> for forecasting, classification and segmentation (for example, predicting post performance or clustering post themes).</li><li>On a related note, the <strong>dashboard</strong> itself can be expanded upon or modified as new actionable insights become apparent.</li><li>I did not pull in <strong>multimedia data</strong>; some of my posts include images that can be highly relevant to their performance. Neither did I ingest my <strong>LinkedIn articles</strong> associated with some of my posts, even though article-level metrics and metadata can give important context to post performance.</li><li><strong>Generative AI</strong> was barely touched on in this blog series, but there is a substantial scope for integrating it into the data product. Databricks offers strong support here, from <strong>Genie spaces</strong> and the <strong>Databricks One</strong> interface for ‘chatting’ with your data, to a wide range of <strong>foundational models</strong> you can deploy for tasks such as content summarisation or recommendation.</li></ul><h4>c. Potential blog series improvements</h4><ul><li>Some of my posts were <strong>code-heavy</strong>, which may be intimidating to our budding data engineers. I plan an open-source repository to make it easier to follow along and reuse the code patterns; more on that later.</li><li>Related to that, several posts make for <strong>heavy reading</strong> due to length and/or technical density. There is potentially room for <strong>tighter editing</strong> to make the writing more <strong>cogent</strong> and <strong>accessible</strong>.</li></ul><h3>IV. Review: Objectives</h3><p>So did I meet the objectives set out at the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">very beginning</a> of the series? I’d say yes, but let’s review.</p><h4>a. To <strong>better understand</strong> my LinkedIn statistics</h4><p>I have indeed achieved a better understanding of my LinkedIn statistics; for instance, I can see the impact of days of the week and posting frequency over a longer time horizon and at a per-post level, in a way that the native LinkedIn dashboard does not easily surface.</p><blockquote>Objective met: <strong>Yes</strong></blockquote><h4>b. To <strong>guide others</strong> in analyzing their own LinkedIn statistics</h4><p>This comprehesive guide will, I hope, help those among you keen to analyse your own LinkedIn statistics. If you are less technical and want something more plug and play, the planned open-source repository will include instructions to get you started.</p><blockquote>Objective met: <strong>Yes (and will be strengthened once the open-source repository is live)</strong></blockquote><h4>c. To <strong>further explore Databricks</strong>’ capabilities and limitations</h4><p>This objective has been thoroughly met, as reflected in the personal learnings and technical limitations discussed earlier in this post.</p><blockquote>Objective met: <strong>Yes</strong></blockquote><h4>d. To <strong>provide a reference end-to-end data product</strong> for budding data professionals</h4><p>Once the open-source repository is released, this objective can be considered thoroughly met; even now, this blog series already serves as a comprehensive reference, with an end-to-end walkthrough and various tips and tricks of the trade.</p><blockquote>Objective met: <strong>Yes (and will be strengthened once the open-source repository is live)</strong></blockquote><h3>V. What’s Next?</h3><p>In the next and final post of this series, I will outline my future follow-ups, as well as launch a public Git repository with the complete demonstration code that was built throughout the series.</p><p>If you’ve built your own variant or plan to, I’d love to hear what worked, what broke, and what you’d like to see next.</p><p>Until then, stay tuned.</p><p>Yingzhao Ouyang is an AI and data engineering specialist with a distinctive blend of humanities, business, and technical expertise, bringing a uniquely holistic perspective to enterprise data challenges that others with purely technical backgrounds miss. To find out more, visit his website at <a href="https://www.yzouyang.com/">https://www.yzouyang.com/</a> or follow his LinkedIn profile at <a href="https://www.linkedin.com/in/yzouyang/">https://www.linkedin.com/in/yzouyang/</a></p><p>This post is also available on Yingzhao Ouyang’s website: <a href="https://www.yzouyang.com/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned/">https://www.yzouyang.com/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d08f03ae1a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own LinkedIn Analytics Part 10: Observing the Pipeline]]></title>
            <link>https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/72841b55fefd</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[data-engineering-101]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 17:47:28 GMT</pubDate>
            <atom:updated>2026-02-14T03:31:46.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*bmNtGE6Dis1x24OQdT2xGw.png" /></figure><p>By the end of the previous post, we had built our data pipeline for LinkedIn analytics from <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">ingestion</a> all the way to the final <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">dashboard product</a>, as well as put in place <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">proper orchestration and automation</a> together with <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">high maintainability</a>.</p><p>The final step is observability: making sure that we know what’s going on with our pipelines.</p><h3>Article Series</h3><ol><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">Why It Matters (and What You’ll Learn)</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">Choosing the Data Source</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-3-creating-the-data-architecture-287ef0a81f85">Creating the Data Architecture</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">Ingesting Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-5-cleaning-and-transforming-data-7478352d73c8">Cleaning and Transforming Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-6-modelling-the-data-9ea2daa78bcf">Modelling the Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">Dashboard Design for Insights and Impact</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">Orchestrating and Automating the Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">Making a Maintainable Pipeline</a></li><li><strong><em>You are here </em></strong>➡️Observing the Pipeline</li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned-4d08f03ae1a8">Key Takeaways and Lessons Learned</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-12-whats-next-open-sourcing-and-community-edd57f3cea20">What’s Next? Open Sourcing and Community</a></li></ol><h3>TL;DR</h3><ul><li>Observability turns this LinkedIn analytics pipeline from a <strong>black box </strong>into a <strong>predictable data product </strong>with clear Service Level Agreements (SLAs) for reliability.</li><li><strong>Operational observability</strong> ensures you know <strong><em>when</em></strong> jobs and tasks fail, <strong><em>who</em></strong> needs to act, and <strong><em>how</em></strong> alerts flow via Databricks notifications.</li><li><strong>Data observability</strong> tracks <strong>quality dimensions </strong>like completeness, timeliness and uniqueness using Databricks’ built-in profiling, so you <strong>catch silent failures</strong>, not just broken runs.</li><li>For a single-creator LinkedIn stack, Databricks’ native observability is sufficient; platform-neutral stacks (OpenTelemetry, Prometheus, Grafana) pay off once you’re coordinating many pipelines across teams and platforms.</li></ul><h3>I. Why Observability is Important</h3><p>One of the critical aspects of an enterprise-ready data product is reliability. That means the pipeline needs to work most of the time and any issues should be resolved within a certain time period. These details are usually defined by a Service Level Agreement (SLA)<em>, </em>which formalises how reliable the pipeline needs to be and how quickly incidents must be resolved.</p><blockquote>Batch pipelines like ours typically need data ready for consumption by a fixed time or within a defined duration; streaming pipelines add uptime targets (e.g., 99.5%) and tighter SLAs.</blockquote><p>While it would be nice for our pipeline to work 100% of the time, in reality there will be disruptions or errors for any number of reasons. How do we make sure such issues are resolved in a timely manner?</p><ol><li><strong>The appropriate person needs to be alerted.</strong> In smaller setups this would typically be the developer or maintainer of the data product; larger teams would have dedicated ‘Level 1’ (L1) support staff who would triage the most common issues, and escalate to higher levels only if the issue requires a deeper investigation or complex fix.</li><li><strong>The issue needs to be logged and traceable.</strong> Error logs need to clearly indicate where something has gone wrong with associated information; meanwhile, we need to be able to trace which task in which job had the issue. If you can’t see where it failed, you can’t promise when it’ll be fixed.</li></ol><p>If the second point sounds familiar, that’s because we’ve already seen this in action when testing our <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">orchestrated pipeline</a>. The same set of skills and tools needed for debugging come into play when triaging a pipeline, and that includes knowing where to look for pipeline monitoring information as well as writing appropriate logging code within our pipeline.</p><p>If we properly document the code and annotate the logging, we should have sufficient information to at least begin to tackle whatever issue may come up.</p><p>But what are the issues that need to be tracked and addressed?</p><h4>a. Operational health</h4><p>This is the most visible issue that we need to address. If any part of our pipeline stops working, we need to know about and resolve the issue in a timely manner. If your business use case (in this instance, perhaps a weekly LinkedIn review) depends on this data product, every failed run is a missed opportunity to learn and adjust your business strategy (or in this instance, content strategy) in a timely fashion.</p><p>Other than the health of the data pipelines and dashboards, the health of our data infrastructure is also important, even if we can’t necessarily do anything about it. For instance, what’s our recourse if the entire AWS us-east-2 region goes down and takes the entire Databricks Free Edition stack (which is what we’ve been using for this series) down with it?</p><blockquote>This isn’t a hypothetical issue: in October 2025, only a few months before the writing of this post, AWS suffered a major outage and <a href="https://www.linkedin.com/pulse/when-cloud-goes-dark-lessons-from-aws-outage-2025-yingzhao-ouyang-omeue?lipi=urn%3Ali%3Apage%3Ad_flagship3_series_entity%3B9Z8glxCiSr23tQlOCdJk%2Fw%3D%3D">took major websites and services down with it</a>. That was just the hardest-hitting of the outages that followed (on Azure and on Cloudflare).</blockquote><p>If the pipeline is critical enough, we’d implement some form of multi-zonal or multi-region setup for our data infrastructure. This feature on Databricks is only available at the Enterprise level, with the associated costs. Higher resiliency almost always results in higher operational costs, and you will need to evaluate how much that higher resiliency and reliability is worth.</p><h4><strong>b. Data health (aka data quality)</strong></h4><p>Just because the pipeline is running doesn’t mean that the data is correct. We need to establish metrics to measure data health i.e. data quality.</p><p>We used exactly such metrics (as defined by Databricks) to <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e?postPublishedType=repub">evaluate our data sources</a> all the way back in part 2 of our blog series. Let’s recap:</p><ul><li><strong>Consistency:</strong> Data values should not conflict with other values across data sets</li><li><strong>Accuracy:</strong> There should be no errors in the data</li><li><strong>Validity:</strong> The data should conform to a certain format</li><li><strong>Completeness:</strong> There should be no missing data</li><li><strong>Timeliness:</strong> The data should be up to date</li><li><strong>Uniqueness:</strong> There should be no duplicates</li></ul><p>These are the dimensions that we watch over time to spot when something silently goes wrong. We defined Relevance as another metric when we were evaluating our data sources, but that’s no longer necessary in our operational state (we already chose our data to be relevant, unless the data product changes significantly).</p><h3>II. Our Observability Stack</h3><p>Building a data product on a single mature data platform allows us to centralise and use the native observability stack of that platform. In other setups, there can be more than one orchestrator and/or data platform. This can happen when:</p><ol><li>Multiple legacy systems remain from earlier design decisions and migrations were never prioritised (‘don’t fix what’s not broken’).</li><li>Different teams own their own stacks and have independently chosen tooling.</li><li>Parts of the system run on third-party platforms outside of the main data platform.</li></ol><p>In these cases, there would be a greater call for a unified observability stack. A combination of OpenTelemetry (an open standard for emitting traces, metrics and logs), Prometheus (an open-source time-series database) and Grafana (a real-time dashboard) is commonly used as a platform-neutral observability stack; the respective cloud platforms also have their own solutions (e.g. Google Cloud Monitoring/Logging as well as Dataplex).</p><p>For a single-creator LinkedIn analytics stack, Databricks’ native observability is enough; the OpenTelemetry/Prometheus/Grafana route usually pays off when you are coordinating many pipelines across teams or platforms.</p><h4>a. Operational observability</h4><p>We have already seen what the operational monitoring on Databricks looks like when testing our <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">orchestrated pipeline</a>. But what about operational alerting?</p><p>Databricks has a notifications feature linked to its jobs and pipelines, which can be added either on the Databricks UI or as a configuration in a Databricks Asset Bundle (DAB). Here’s a breakdown on when to use which</p><ul><li><strong>Job or pipeline notifications</strong> ensure you hear about any failed run without checking the Jobs UI.</li><li><strong>Task notifications</strong> let you focus alerts on the most critical steps and avoid noise from skipped or cancelled tasks.</li></ul><p>Now let’s see how to enable them for Jobs and Tasks on the Databricks UI; we will explore the DABs approach afterwards.</p><p>Navigate to the interface of one of your Jobs and find the ‘Job notifications’ section as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*jzju2NzxERkcfWjRW-y2jA.png" /><figcaption>The Job notifications section of Databricks Jobs.</figcaption></figure><p>Click on ‘Edit notifications’, and you are brought to the ‘Job notifications’ dialog box. Clicking on ‘Add notification’ then brings you to the setup page for Job notifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*B76A8Fcpyr90K5AfFZrqdA.png" /><figcaption>Dialog box for job notifications.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*YYDMuvbzBzOz80P9Vb8tDw.png" /><figcaption>Setup dialog box for job notifications. Note the checkboxes for different states of the job run; ‘Failure’ is ticked by default. Note also the mute options for skipped or cancelled runs.</figcaption></figure><p>The default destination type for notifications is an email address, as can be seen in the screencap. However, other system destinations can be set up, such as to a Microsoft Teams channel or to a webhook.</p><p>For our purposes, email alerts are perfectly fine, though you could set up a webhook to your Telegram, WhatsApp or other messaging system of your choice if you prefer more immediate notifications than an email.</p><p>Below are some screencaps of what an email notification setup looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*13CfXK9kTDu07K3hdzKwzw.png" /><figcaption>The job notifications section of Databricks Jobs, now with a single email notification on failure.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*iHDIlllwMRLKhpTYJ7qbAg.png" /><figcaption>Partial screencap of an email that reports the successful signup for email notifications to a particular Databricks Job.</figcaption></figure><p>Accessing task notifications is a similar story. There is a ‘Notifications’ section in the task setup, and clicking ‘+ Add’ brings us to a similar ‘Task notifications’ dialog box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*9PBiNKmS9UxevQEm-hirHw.png" /><figcaption>The ‘Notifications’ section of the Databricks Jobs task setup.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*jfEcyE67TRJ1TZk4mIBHAg.png" /><figcaption>The dialog box for task notifications.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*q483StwbDMJ9BblU9beVYg.png" /><figcaption>The setup dialog box for task notifications. There is an extra mute option for ignoring all but the final retry of a task.</figcaption></figure><p>Once you’re happy with the UI setup, you can replicate the same notifications into a DAB so they’re versioned and reproducible (i.e. <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a">maintainable</a>):</p><pre>resources:<br>  jobs:<br>    job_name:<br>      name: ...<br>      email_notifications:<br>        on_failure:<br>          - placeholder@email.com<br>        no_alert_for_skipped_runs: true<br>      notification_settings:<br>        no_alert_for_skipped_runs: true<br>        no_alert_for_canceled_runs: true<br>      ...<br>      tasks:<br>        - task_key: task_name_1<br>          ...<br>          email_notifications:<br>            on_failure:<br>              - placeholder@email.com<br>          notification_settings:<br>            no_alert_for_skipped_runs: true<br>            no_alert_for_canceled_runs: true<br>            alert_on_last_attempt: true<br>        - task_key: task_name_2<br>          ...</pre><p>If you’re setting up custom notification systems, note that these cannot yet be set up via DABs, but you can still direct the notifications to a webhook; its configuration can be stored in a DAB variable.</p><p>Similar notification setups can be done for Declarative Pipelines, but they are not as extensive, as can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*o3b_WDgxIzAEwJ6f84mZtg.png" /><figcaption>Notification setup for pipelines; note that they can be only be set at the pipeline rather than task level, and they can only be sent via email.</figcaption></figure><h4>b. Data observability</h4><p>Earlier, we defined data quality dimensions like completeness, timeliness and uniqueness. In observability terms, these become checks and dashboards that tell us when the numbers we rely on for LinkedIn decisions might be wrong.</p><p>We already defined what constitutes good data quality for our data when <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">choosing our data sources</a>. Generally speaking, we don’t stop our pipelines on data quality issues unless such issues break our pipelines (e.g. unexpected data schema changes). However, we still want to ensure that we can monitor such issues as and when they appear, and we would like to be alerted if the quality issues cross a certain threshold.</p><p>Databricks has a data quality feature built into its tables. Click on any table and navigate to the ‘Quality’ tab; you’ll see the option to enable Data Quality Monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Sxdqd_7jBq_paH0GCfIRw.png" /><figcaption>Example screencap of Databricks’ native data quality monitoring configuration.</figcaption></figure><p>Clicking on ‘Enable’ shows us two options: a schema-wide ‘intelligent’ anomaly detection feature, and data profiling for column and row-level metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*Lk74Hyat4nFTRTRb7DdyeQ.png" /><figcaption>Choices for Databricks’ native data quality monitoring.</figcaption></figure><p>The data profiling feature is where you can set up Databricks’ more generic data quality monitoring. There are three main options, as seen in the below screencap:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*rVcGRvYJurUo-_-pclym4Q.png" /><figcaption>Three types of data profiling available.</figcaption></figure><ol><li><strong>Time series profiling</strong> analyses quality metrics over a set of time windows. In our case, daily time windows make sense for much of our data, e.g. our impressions and engagements silver tables.</li><li><strong>Snapshot profiling</strong> analyses quality metrics over the entire table and is used when there is no useful or relevant time field to be used, e.g. for our posts silver tables. Note that materialized tables can only use snapshot profiling.</li><li><strong>Inference profiling</strong> is specifically for tracking machine learning model drift and performance over time. Since we’re not running any such models, this is not relevant for us. If we decide to deploy machine learning models later (e.g. forecasting model for impressions), enabling this feature will help us catch any underlying drift in the data that can result in a reduction in the models’ accuracy.</li></ol><p>There is also an advanced options section where refresh schedules, the location of data profile metric tables, as well as optional custom metrics can be defined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*yM7wlgYixLe0FQKBUFGV-w.png" /><figcaption>Advanced options for data quality monitor creation.</figcaption></figure><p>Once the data profiling has been configured and saved, a dashboard with its two associated metrics tables is created, see the screencaps below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*XJHoRtjb_tC-bPWPtiESTQ.png" /><figcaption>Data profiling configured. Note that the dashboard will have no data until the extra supporting tables have been created by Databricks.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*QgEq2GdmK5GTUgaj2Ao3nw.png" /><figcaption>The two extra tables created by data profiling.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWOCgYTak-pzs6sv9Dvgnw.png" /><figcaption>Example screencap of time series data profiling.</figcaption></figure><p>Finally, all of the above data profiling can be configured via DABs, as per Databricks’ <a href="https://docs.databricks.com/aws/en/dev-tools/bundles/resources#quality_monitor-unity-catalog">documentation</a>.</p><blockquote>As of the time of writing, anomaly detection cannot be set up via DABs; if you want to use a CI/CD approach, you will have to explore either the <a href="https://docs.databricks.com/api/workspace/dataquality/createmonitor#anomaly_detection_config-excluded_table_full_names">Data Quality Monitoring API</a> or <a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/data_quality_monitor">the equivalent Terraform module</a>.</blockquote><h3>III. What’s Next?</h3><p>Our journey to build our LinkedIn analytics data product is almost complete. Next up is a post-mortem where I share key takeaways and lessons learned when working on this project, so stay tuned.</p><p>Yingzhao Ouyang is an AI and data engineering specialist with a distinctive blend of humanities, business, and technical expertise, bringing a uniquely holistic perspective to enterprise data challenges that others with purely technical backgrounds miss. To find out more, visit his website at <a href="https://www.yzouyang.com/">https://www.yzouyang.com/</a> or follow his LinkedIn profile at <a href="https://www.linkedin.com/in/yzouyang/">https://www.linkedin.com/in/yzouyang/</a></p><p>This post is also available on Yingzhao Ouyang’s website: <a href="https://www.yzouyang.com/build-your-own-linkedin-analytics-part-10-observing-the-pipeline/">https://www.yzouyang.com/build-your-own-linkedin-analytics-part-10-observing-the-pipeline/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72841b55fefd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own LinkedIn Analytics Part 9: Making a Maintainable Pipeline]]></title>
            <link>https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline-d2c35a874e3a?source=rss-f4140c532c0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2c35a874e3a</guid>
            <category><![CDATA[data-engineering-101]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Yingzhao Ouyang]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 01:02:22 GMT</pubDate>
            <atom:updated>2026-01-19T18:57:44.401Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*eIVfoeBgIWPZLW2EuQA9uA.png" /></figure><p>By the end of the previous post, we had built our data pipeline for LinkedIn analytics from <a href="http://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">ingestion</a> all the way to the final <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">dashboard product</a>, as well as put in place <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">proper orchestration and automation</a> of our data product.</p><p>The next step then is making sure that all of what we’ve just built is easy to maintain.</p><h3>Article Series</h3><ol><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-1-why-it-matters-and-what-youll-learn-314a4de63988">Why It Matters (and What You’ll Learn)</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-2-choosing-the-data-source-fa0b08c67d8e">Choosing the Data Source</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-3-creating-the-data-architecture-287ef0a81f85">Creating the Data Architecture</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-4-ingesting-data-21517387a1b7">Ingesting Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-5-cleaning-and-transforming-data-7478352d73c8">Cleaning and Transforming Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-6-modelling-the-data-9ea2daa78bcf">Modelling the Data</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-7-dashboard-design-for-insights-and-impact-ab987fd05f3a">Dashboard Design for Insights and Impact</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">Orchestrating and Automating the Pipeline</a></li><li><strong><em>You are here </em></strong>➡️Making a Maintainable Pipeline</li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-10-observing-the-pipeline-72841b55fefd">Observing the Pipeline</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-11-key-takeaways-and-lessons-learned-4d08f03ae1a8">Key Takeaways and Lessons Learned</a></li><li><a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-12-whats-next-open-sourcing-and-community-edd57f3cea20">What’s Next? Open Sourcing and Community</a></li></ol><h3>TL;DR</h3><ul><li>Use Git and Databricks Asset Bundles to turn your LinkedIn analytics pipeline from a Databricks POC into a versioned, reproducible project with dev/prod separation, safe rollbacks, and one‑click deployments.</li></ul><h3>I. Options for Maintainability</h3><p>A maintainable pipeline is one that has the following qualities:</p><ul><li>Easy to change;</li><li>Easy to roll back; and</li><li>Safe to run autonomously without manual babysitting.</li></ul><p>You can thus safely introduce changes such as new engagement metrics, knowing you can roll back if something breaks. For a data‑driven LinkedIn strategy, that reliability is what lets you experiment aggressively without losing trust in your numbers.</p><p>A lot of what makes a pipeline maintainable is already part of our design for our LinkedIn analytics data product.</p><ol><li><strong>Well-documented assets:</strong> This includes readable and well-structured code, clear comments, and relevant descriptions of data assets.</li><li><strong>Modular notebooks as tasks: </strong>The notebooks that make up our task nodes can be swapped out for code that does the equivalent; there is no dependency on any hidden state or special flag.</li><li><strong>Idempotent tasks and runs:</strong> We can re-run a failed task or pipeline without worrying about data consistency.</li><li><strong>Data versioning via Delta Lake tables:</strong> Databricks managed tables have versioning enabled by default, so any errors committed during an update can be rolled back with little issue.</li></ol><p>But we need some additional qualities:</p><ol><li><strong>Version control for notebooks, jobs, dashboards and schema definitions:</strong> If a pipeline were to be tampered with or erroneous code introduced, we have no easy way of restoring the previous state.</li><li><strong>Separation of development from production:</strong> We want to have a version of the pipeline and assets that can be used for testing new features or fixes (development), before the changes are pushed to the main pipeline (production).</li><li><strong>Continuous integration and deployment:</strong> With this distinction between development and production, we also want to have a way to manage all these differences in a controlled and automated fashion.</li></ol><p>We have two main ways of achieving these qualities, which we will be using in combination.</p><h4>a. Git repositories</h4><p><a href="https://git-scm.com/book/en/v2/Getting-Started-What-is-Git">Git repositories</a> are the modern way of managing code versions, and Databricks does support them out of the box. The problem is that we don’t yet have a code version of our Databricks infrastructure such as our Databricks Jobs or our dashboard. How do we version those?</p><h4>b. Databricks Asset Bundles</h4><p>Databricks introduced this feature in 2024 and has been <a href="https://docs.databricks.com/aws/en/release-notes/dev-tools/bundles">iterating on it ever since</a>. As the name suggests, this bundles Databricks assets together in code form, including Jobs, Dashboards and even schemas. In other words, this is Databricks’ version of Infrastructure as Code (IaC). This concept was popularized by HashiCorp Terraform, and indeed Databricks Asset Bundles (or DABs) were initially built on top of Terraform, though there is now a push to move away from that dependency.</p><p>Note that DABs are meant to be used together with Git repositories; in fact, in the Databricks UI, you can only create a DAB in a Git repository folder.</p><h3>II. Setting up Git in Databricks</h3><p>This step assumes that you already have a Git repository at hand. Databricks supports a comprehensive array of Git providers, including the various flavours of GitHub, GitLab, BitBucket, Azure DevOps and AWS CodeCommit. For our purposes, GitHub works just fine.</p><p>In the Databricks UI, click on your profile icon and choose the ‘Settings’ option. You should see the following menu on the left:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*_L-mb9ehUWuigcPGOfzqjA.png" /><figcaption>Setting menu for Databricks.</figcaption></figure><p>Click on ‘Linked accounts’ under the ‘User’ section. As of this writing the only service available for connection is Git integration, which is exactly what we want. Click on ‘Add Git credentials’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*5rqdkSUKeXVsCTKwfGXYkQ.png" /><figcaption>The ‘Linked accounts’ settings.</figcaption></figure><p>Here you choose your Git provider. If you choose GitHub, you can then link your Git account in a few clicks. You can also opt to use a personal access token if you don’t wish to publish your associated email to the repository (which may be a concern if you intend to connect to or create a public repository). Other Git providers require you to provide similar access tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*90VuG_TRRh-H-khlmIn4tA.png" /><figcaption>The ‘Add Git credential’ dialog box, with GitHub selected as the Git provider.</figcaption></figure><p>The next step is to create a Git folder in our workspace. Click ‘+ New’ at the top left corner, navigate to ‘More’, and then choose ‘Git folder’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*KgZJPkVlHm-6B_uhycLmcg.png" /><figcaption>Navigating to the ‘Git folder’ option.</figcaption></figure><p>You can now create your Git folder. Obtain your Git repository URL (the steps to obtain it for GitHub are <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository">here</a>), name the folder, then proceed to create it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*aFNd5H2jVc8c1IuJP-se9w.png" /><figcaption>Dialog box for creating a Git folder.</figcaption></figure><p>Below is the end result of this process: a Git folder on Databricks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*2h7OCny4aP_q1WUgqkJBLQ.png" /><figcaption>Example of a Git folder in Databricks, which also indicates the Git branch that the folder is currently pointing to.</figcaption></figure><p>Git is a fundamental tool for code versioning, and is a required competency for any aspiring data engineer (and developer in general). If you want to dive deeper, there is a comprehensive guide <a href="https://git-scm.com/book/en/v2">here</a>.</p><h3>III. Setting up a service principal</h3><p>If you look at the ‘Run as’ column in Jobs and Pipelines, you can see that all our runs so far have been using our personal credentials. This is fine for development work, but for a production run, that is misleading at best (we aren’t doing a manual run) and potentially catastrophic at worst (what happens when the pipeline developer leaves the organization?).</p><blockquote>In general, we should always use synthetic credentials for production runs and deployments, whether they come in the form of service principals or service accounts.</blockquote><p>With that in mind, let’s create a Databricks service principal. Back in the ‘Settings’ menu, click on the ‘Identity and access’ option under ‘Workspace admin’. You will see the following section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*-3SQpOhOoXy26MvyKvTT-A.png" /><figcaption>Options for identity and access in a Databricks workspace.</figcaption></figure><p>Administering users and groups is beyond the scope of our exploration, especially in our Databricks Free instance, but it is important to set these up for any enterprise deployment. Databricks has some recommended best practices in their <a href="https://docs.databricks.com/aws/en/admin/users-groups/best-practices">documentation</a>.</p><p>Click on ‘Manage’ for Service principals, then ‘Add service principal’ on the next page, and finally ‘Add new’ in the subsequent page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6YIH1p4HuwhpYG3O_kje8A.png" /><figcaption>The Service principals page.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*UVgNjLZ9Gn-dSd0MfhXMOQ.png" /><figcaption>The ‘Add service principal’ dialog box.</figcaption></figure><p>Type a descriptive name for the service principal, then click ‘Add’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*vMSLvNxROkx4_n8s1hLwpg.png" /><figcaption>The ‘Add service principal’ dialog box with a service principal about to be created.</figcaption></figure><p>Note the Application ID of our newly created service principal; we will be using it later in our DAB deployment.</p><h3>IV. Creating our Databricks Asset Bundle</h3><p>Navigate to our Git folder, and click ‘Create’ at the top right corner. You will see ‘Asset bundle’ as one of the options; click on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*UIOvqTQBKSftFog2EAaNmQ.png" /><figcaption>‘Create’ options available in a Git folder. Note that ‘Asset bundle’ only appears as an option in a Git folder.</figcaption></figure><p>This brings you to the list of options for creating a Databricks Asset Bundle as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*YVTidrKPRUNuj3CzQQYMxw.png" /><figcaption>Options for creating an asset bundle in Databricks.</figcaption></figure><p>For the purposes of this project, choose ‘Empty project’; you can explore the more guided options separately.</p><p>This is a good time to outline what our structure for our DAB is going to look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*rM7A84TmUz6i5eX6w5r6sw.png" /><figcaption>Outline of structure for the DAB for our LinkedIn analytics data product.</figcaption></figure><ol><li>There are a bunch of additional YAML files in the ‘resources’ folder. This is best practice for separation of different types of resources; we don’t want to dump everything into a single <strong>databricks.yml</strong> file, for the same reason we don’t want to have a monolithic notebook as our pipeline.</li><li>Note the <strong>‘variable-override.json’</strong> file in the <strong>‘.databricks’</strong> folder. Because this folder is listed in the <strong>‘.gitignore’</strong> file, it and its contents will not be written to the Git repository. This is a good way to store any local configurations that you don’t want to expose in the repository, such as the Databricks warehouse ID and the service principal ID.</li><li>The notebooks from our POC pipeline have been copied into this DAB. I have arranged them in numbered folders under ‘src’ (aka the source folder). I have also included the POC notebook that was the source for the individual notebook tasks; this can prove useful in trying out end-to-end ideas in the development stage.</li><li>The gold layer notebooks are an exception, having been replaced with SQL files instead as part of a Spark Declarative Pipeline. We touch on this further when covering the pipeline.yml configuration file.</li></ol><p>Let’s go through the different YAML configuration files in turn.</p><h4>a. variables.yml</h4><p>This file serves as a single place for defining our pipeline-wide constants. With this, we no longer have to repeat them through multiple notebooks, or manually keep the naming of data assets such as schemas and tables consistent.</p><p>The contents of the file are as below:</p><pre>variables:<br><br>  # 0. Sensitive variables<br>  linkedin_profile_name:<br>    description: &gt;<br>      The name of the LinkedIn profile<br>  warehouse_id:<br>    description: &gt;<br>      The ID of the Databricks SQL warehouse cluster<br>  pipeline_runner:<br>    description: &gt;<br>      The name of the service principal used to deploy <br>      the DAB and run the pipelines<br><br>  # 1. Settings for the medallion layer catalogs and schemas<br>  landing_catalog: <br>    description: &gt;<br>      The ID of the landing catalog<br>    default: landing<br>  bronze_catalog: <br>    description: &gt;<br>      The ID of the bronze catalog<br>    default: bronze<br>  silver_catalog: <br>    description: &gt;<br>      The ID of the silver catalog<br>    default: silver<br>  gold_catalog: <br>    description: &gt;<br>      The ID of the gold catalog<br>    default: gold<br>  common_schema:<br>    description: &gt;<br>      The ID of the schema used in all medallion catalogs<br>    default: linkedin<br><br>  # 2. Settings for the landing volumes<br>  landing_content_daily_volume:<br>    description: &gt;<br>      The ID of the landing volume for <br>      daily content metrics Excel files<br>    default: content_daily<br>  landing_content_historical_volume:<br>    description: &gt;<br>      The ID of the landing volume for <br>      historical content metrics Excel files<br>    default: historical_daily<br>  landing_posts_volume:<br>    description: &gt;<br>      The ID of the landing volume for <br>      post metrics Excel files<br>    default: posts<br>  landing_patch_volume:<br>    description: &gt;<br>      The ID of the landing volume for <br>      patch CSV files<br>    default: patch<br>  ingestion_folder:<br>    description: &gt;<br>      The folder in each volume for <br>      uploading files to be ingested<br>    default: pending<br>  processed_folder:<br>    description: &gt;<br>      The folder in each volume for <br>      storing successfully ingested files<br>    default: processed<br>  errors_folder:<br>    description: &gt;<br>      The folder in each volume for storing files <br>      with errors during ingestion<br>    default: errors<br>  patch_post_subfolder:<br>    description: &gt;<br>      The subfolder in the landing patch volume for <br>      patching posts data<br>    default: posts<br><br>  # 3. Settings for the bronze tables<br>  bronze_profile_discovery_metrics_table:<br>    description: &gt;<br>      The ID of the bronze table for <br>      profile daily discovery metrics<br>    default: discovery<br>  bronze_profile_followers_table:<br>    description: &gt;<br>      The ID of the bronze table for <br>      profile daily follower metrics<br>    default: followers<br>  bronze_post_patch_table:<br>    description: &gt;<br>      The ID of the bronze table for post patching<br>    default: linkedin_patch<br>  bronze_post_metrics_table:<br>    description: &gt;<br>      The ID of the bronze table for post metrics <br>      (taken from post metrics Excel files)<br>    default: post_details<br>  bronze_post_metadata_table:<br>    description: &gt;<br>      The ID of the bronze table for post metadata<br>    default: posts<br>  bronze_profile_metrics_table:<br>    description: &gt;<br>      The ID of the bronze table for profile daily engagement <br>      and impressions metrics<br>    default: totals<br><br>  # 3.1 Settings for the bronze staging tables<br>  bronze_staging_impressions_prefix:<br>    description: &gt;<br>      The prefix for bronze staging tables for <br>      daily post impressions<br>    default: impressions<br>  bronze_staging_engagements_prefix:<br>    description: &gt;<br>      The prefix for bronze staging tables for <br>      daily post engagements<br>    default: engagements<br><br>  # 4. Settings for the silver tables<br>  silver_post_impressions:<br>    description: &gt;<br>      The ID of the silver table for post impressions<br>    default: impressions<br>  silver_post_engagements:<br>    description: &gt;<br>      The ID of the silver table for post engagements<br>    default: engagements<br>  silver_profile_followers:<br>    description: &gt;<br>      The ID of the silver table for profile follower metrics<br>    default: followers<br>  silver_post_metadata:<br>    description: &gt;<br>      The ID of the silver table for consolidated post metadata<br>    default: posts<br>  silver_profile_metrics:<br>    description: &gt;<br>      The ID of the silver table for profile engagement <br>      and impressions metrics<br>    default: totals<br><br>  # 4. Settings for the gold tables<br>  gold_fact_daily_post_statistics_table:<br>    description: &gt;<br>      The ID of the gold fact table/view for daily posts statistics<br>    default: fct_daily_post_statistics<br>  gold_fact_daily_profile_statistics_table:<br>    description: &gt;<br>      The ID of the gold fact table/view for daily profile statistics<br>    default: fct_daily_profile_statistics<br>  gold_dimension_date_table:<br>    description: &gt;<br>      The ID of the gold dimension table/view for dates<br>    default: dim_date<br>  gold_dimension_timestamp_table:<br>    description: &gt;<br>      The ID of the gold dimension table/view for timestamps<br>    default: dim_timestamp<br><br>  # 5. Settings for the dashboard<br>  dashboard_subscriber:<br>    description: &gt;<br>      The email of the subscriber to the dashboard<br>    default: ${workspace.current_user.userName}</pre><p>Now changing a table name or adding a new volume is a single variable change, not a hunt through multiple notebooks and jobs.</p><p>For the variables with no defaults defined due to their sensitivity, they are defined in the override JSON located in the <strong>.databricks</strong> folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*pALLEm02-deOqtTFr0gIRg.png" /><figcaption>Location of the variable overrides JSON file.</figcaption></figure><p>The structure of the file is as follows.</p><pre># variable-overrides.json<br>{<br>  &quot;linkedin_profile_name&quot;: &quot;placeholder&quot;,<br>  &quot;warehouse_id&quot;: &quot;placeholder&quot;,<br>  &quot;pipeline_runner&quot;: &quot;placeholder&quot;<br>}</pre><h4>b. schemas.yml</h4><p>We define our schemas as well as our volumes here:</p><pre>resources:<br>  schemas:<br>    landing:<br>      catalog_name: ${var.landing_catalog}<br>      comment: LinkedIn landing schema created by DAB<br>      name: ${var.common_schema}<br>    bronze:<br>      catalog_name: ${var.bronze_catalog}<br>      comment: LinkedIn bronze schema created by DAB<br>      name: ${var.common_schema}<br>    silver:<br>      catalog_name: ${var.silver_catalog}<br>      comment: LinkedIn silver schema created by DAB<br>      name: ${var.common_schema}<br>    gold:<br>      catalog_name: ${var.gold_catalog}<br>      comment: LinkedIn gold schema created by DAB<br>      name: ${var.common_schema}<br>  volumes:<br>    content_daily:<br>      catalog_name: ${var.landing_catalog}<br>      schema_name: ${resources.schemas.landing.name}<br>      name: ${var.landing_content_daily_volume}<br>    content_historical:<br>      catalog_name: ${var.landing_catalog}<br>      schema_name: ${resources.schemas.landing.name}<br>      name: ${var.landing_content_historical_volume}<br>    posts:<br>      catalog_name: ${var.landing_catalog}<br>      schema_name: ${resources.schemas.landing.name}<br>      name: ${var.landing_posts_volume}<br>    patch:<br>      catalog_name: ${var.landing_catalog}<br>      schema_name: ${resources.schemas.landing.name}<br>      name: ${var.landing_patch_volume}</pre><p>Here we see our variables in action in deploying consistent namings for schemas and volumes that are also fed into our notebooks.</p><h4>c. dashboard.yml</h4><p>We define our dashboard as below:</p><pre>sync:<br>  exclude:<br>    - ../src/*.lvdash.json<br>    - ../src/*/*.lvdash.json<br>resources:<br>  dashboards:<br>    linkedin_statistics:<br>      display_name: LinkedIn Statistics<br>      file_path: &gt;<br>        ../src/linkedin_analytics_jobs/<br>        4. data product/LinkedIn Statistics.lvdash.json<br>      warehouse_id: ${var.warehouse_id}</pre><p>The exclusions are to avoid duplication of dashboards, per Databricks’ <a href="https://docs.databricks.com/aws/en/dev-tools/bundles/resources#dashboard">advice</a>.</p><p>As of this writing, there is no straightforward, documented way to pull DAB variables directly into the dataset queries inside a Databricks dashboard, as evidenced by <a href="https://github.com/databricks/cli/issues/1915">an open Databricks CLI GitHub issue</a>. <a href="https://docs.databricks.com/aws/en/dashboards/parameters">Dashboard parameters</a> are instead designed for end-user interactivity via widgets. A workaround using third-party CI/CD tools such as GitHub Actions and Jenkins to template the dashboard JSON before deployment is possible; we touch on such tools towards the end of this article.</p><h4>d. jobs.yml</h4><p>Recall the pipeline structure that we finalized in the <a href="https://medium.com/@kunojilym/build-your-own-linkedin-analytics-part-8-orchestrating-and-automating-the-pipeline-9a64c8d1eed8">previous post</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ku_ly-OdZGTBUmeHIBxX_w.png" /><figcaption>Recap: pipeline structure for our LinkedIn analytics data product</figcaption></figure><p>We want to embed this all in code. But how?</p><p>Navigate to any of the Databricks Jobs. Click the three dots next to the ‘Run now’ button, and choose ‘View as code’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/1*IE-v2FVl1gzUWaaIlFClug.png" /><figcaption>Some of the options available for Databricks Jobs.</figcaption></figure><p>Now we can see the code version of the Databricks Job, which has defaulted to YAML format. We can copy this and paste it into ‘<strong>jobs.yml</strong>’, then edit accordingly (e.g. replacing hard-coded values with our constants). We need to repeat this for all of our pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*ArlpSqTbjPmUucDjlaPSvw.png" /><figcaption>Example of a Databricks Job in YAML form.</figcaption></figure><p>One of the items that needs changing for this Databricks Job is the <strong>job_id </strong>that is triggered at the end of the pipeline. We can substitute an inferred variable in place of the hard-coded number, as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*oyOfoCSf_s-gG-WBYW2AQg.png" /><figcaption>Example of a variable inferred from a different Databricks resource defined in the DAB.</figcaption></figure><p>For more details on the format for the types of tasks that can be added to Databricks Jobs in a DAB, refer to <a href="https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types">this link</a>.</p><p>At the end of this import and refactoring process, we end up with a file that consolidates all our job definitions and uses our defined constants and inferred variables, as per below:</p><pre>resources:<br>  jobs:<br>    linkedin_transformation:<br>      name: linkedin transformation<br>      tasks:<br>        - task_key: silver_engagements_transformation<br>          notebook_task:<br>            base_parameters:<br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_ENGAGEMENTS_TABLE_PREFIX: &gt;<br>                ${var.bronze_staging_engagements_prefix}<br>              BRONZE_TOTALS_TABLE: &gt;<br>                ${var.bronze_profile_metrics_table}<br><br>              SILVER_CATALOG: &gt;<br>                ${var.silver_catalog}<br>              SILVER_SCHEMA: &gt;<br>                ${resources.schemas.silver.name}<br>              SILVER_ENGAGEMENTS_TABLE: &gt;<br>                ${var.silver_post_engagements}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              2. silver transformation/<br>              silver engagements consolidation.ipynb<br>            source: WORKSPACE<br>        - task_key: silver_impressions_transformation<br>          notebook_task:<br>            base_parameters:<br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_IMPRESSIONS_TABLE_PREFIX: &gt;<br>                ${var.bronze_staging_impressions_prefix}<br>              BRONZE_TOTALS_TABLE: &gt;<br>                ${var.bronze_profile_metrics_table}<br><br>              SILVER_CATALOG: &gt;<br>                ${var.silver_catalog}<br>              SILVER_SCHEMA: &gt;<br>                ${resources.schemas.silver.name}<br>              SILVER_IMPRESSIONS_TABLE: &gt;<br>                ${var.silver_post_impressions}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              2. silver transformation/<br>              silver impressions consolidation.ipynb<br>            source: WORKSPACE<br>        - task_key: silver_views_update<br>          notebook_task:<br>            base_parameters:<br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_FOLLOWERS_TABLE: &gt;<br>                ${var.bronze_profile_followers_table}<br>              BRONZE_TOTALS_TABLE: &gt;<br>                ${var.bronze_profile_metrics_table}<br><br>              SILVER_CATALOG: &gt;<br>                ${var.silver_catalog}<br>              SILVER_SCHEMA: &gt;<br>                ${resources.schemas.silver.name}<br>              SILVER_FOLLOWERS_TABLE: &gt;<br>                ${var.silver_profile_followers}<br>              SILVER_TOTALS_TABLE: &gt;<br>                ${var.silver_profile_metrics}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              2. silver transformation/<br>              silver views update.ipynb<br>            source: WORKSPACE<br>        - task_key: silver_posts_enrichment<br>          notebook_task:<br>            base_parameters:<br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_POSTS_TABLE: &gt;<br>                ${var.bronze_post_metadata_table}<br>              BRONZE_POST_DETAILS_TABLE: &gt;<br>                ${var.bronze_post_metrics_table}<br>              BRONZE_POST_PATCH_TABLE: &gt;<br>                ${var.bronze_post_patch_table}<br><br>              SILVER_CATALOG: &gt;<br>                ${var.silver_catalog}<br>              SILVER_SCHEMA: &gt;<br>                ${resources.schemas.silver.name}<br>              SILVER_POSTS_TABLE: &gt;<br>                ${var.silver_post_metadata}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              2. silver transformation/<br>              silver post enrichment.ipynb<br>            source: WORKSPACE<br>        - task_key: gold_create_or_refresh<br>          depends_on:<br>            - task_key: silver_engagements_transformation<br>            - task_key: silver_impressions_transformation<br>            - task_key: silver_posts_enrichment<br>            - task_key: silver_views_update<br>          pipeline_task:<br>            pipeline_id: &gt;<br>              ${resources.pipelines.gold_create.id}<br>          max_retries: 3<br>          min_retry_interval_millis: 60000<br>          retry_on_timeout: true<br>        - task_key: dashboard_refresh<br>          depends_on:<br>            - task_key: gold_create_or_refresh<br>          dashboard_task:<br>            subscription:<br>              subscribers:<br>                - user_name: &gt;<br>                  ${var.dashboard_subscriber}<br>            warehouse_id: &gt;<br>              ${var.warehouse_id}<br>            dashboard_id: &gt;<br>              ${resources.dashboards.linkedin_statistics.id}<br>      queue:<br>        enabled: true<br>      performance_target: PERFORMANCE_OPTIMIZED<br>    linkedin_daily_ingest:<br>      name: linkedin daily ingest<br>      trigger:<br>        pause_status: UNPAUSED<br>        file_arrival:<br>          url: &gt;<br>            /Volumes/${var.landing_catalog}/<br>            ${resources.schemas.landing.name}/<br>            ${resources.volumes.content_daily.name}/<br>            ${var.ingestion_folder}/<br>      tasks:<br>        - task_key: bronze_daily_ingest<br>          notebook_task:<br>            base_parameters:<br>              LINKEDIN_PROFILE_NAME: &gt;<br>                ${var.linkedin_profile_name}<br><br>              LANDING_CATALOG: &gt;<br>                ${var.landing_catalog}<br>              LANDING_SCHEMA: &gt;<br>                ${resources.schemas.landing.name}<br>              LANDING_DAILY_VOLUME: &gt;<br>                ${resources.volumes.content_daily.name}<br><br>              PENDING_FOLDER: &gt;<br>                ${var.ingestion_folder}<br>              PROCESSED_FOLDER: &gt;<br>                ${var.processed_folder}<br>              ERRORS_FOLDER: &gt;<br>                ${var.errors_folder} <br><br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_DISCOVERY_TABLE: &gt;<br>                ${var.bronze_profile_discovery_metrics_table}<br>              BRONZE_TOTALS_TABLE: &gt;<br>                ${var.bronze_profile_metrics_table}<br>              BRONZE_FOLLOWERS_TABLE: &gt;<br>                ${var.bronze_profile_followers_table}<br><br>              BRONZE_IMPRESSIONS_TABLE: &gt;<br>                ${var.bronze_staging_impressions_prefix}<br>              BRONZE_ENGAGEMENTS_TABLE: &gt;<br>                ${var.bronze_staging_engagements_prefix}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              1. bronze ingestion/<br>              bronze daily ingest.ipynb<br>            source: WORKSPACE<br>        - task_key: bronze_post_ingest<br>          depends_on:<br>            - task_key: bronze_daily_ingest<br>          notebook_task:<br>            base_parameters:<br>              LINKEDIN_PROFILE_NAME: &gt;<br>                ${var.linkedin_profile_name}<br><br>              LANDING_CATALOG: &gt;<br>                ${var.landing_catalog}<br>              LANDING_SCHEMA: &gt;<br>                ${resources.schemas.landing.name}<br>              LANDING_POSTS_VOLUME: &gt;<br>                ${resources.volumes.posts.name}<br>              <br>              PENDING_FOLDER: &gt;<br>                ${var.ingestion_folder}<br>              PROCESSED_FOLDER: &gt;<br>                ${var.processed_folder}<br>              ERRORS_FOLDER: &gt;<br>                ${var.errors_folder}<br><br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_IMPRESSIONS_TABLE: &gt;<br>                ${var.bronze_staging_impressions_prefix}<br>              BRONZE_POSTS_TABLE: &gt;<br>                ${var.bronze_post_metadata_table}<br>              BRONZE_POST_DETAILS_TABLE: &gt;<br>                ${var.bronze_post_metrics_table}<br>              BRONZE_POST_PATCH_TABLE: &gt;<br>                ${var.bronze_post_patch_table}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              1. bronze ingestion/<br>              bronze post ingest.ipynb<br>            source: WORKSPACE<br>        - task_key: trigger_transformation<br>          depends_on:<br>            - task_key: bronze_post_ingest<br>          run_job_task:<br>            job_id: &gt;<br>              ${resources.jobs.linkedin_transformation.id}<br>      queue:<br>        enabled: true<br>      performance_target: PERFORMANCE_OPTIMIZED<br>    linkedin_historical_ingestion:<br>      name: linkedin historical ingestion<br>      trigger:<br>        pause_status: UNPAUSED<br>        file_arrival:<br>          url: &gt;<br>            /Volumes/${var.landing_catalog}/<br>            ${resources.schemas.landing.name}/<br>            ${resources.volumes.content_historical.name}/<br>            ${var.ingestion_folder}/<br>      tasks:<br>        - task_key: bronze_historical_ingest<br>          notebook_task:<br>            base_parameters:<br>              LINKEDIN_PROFILE_NAME: &gt;<br>                ${var.linkedin_profile_name}<br><br>              LANDING_CATALOG: &gt;<br>                ${var.landing_catalog}<br>              LANDING_SCHEMA: &gt;<br>                ${resources.schemas.landing.name}<br>              LANDING_DAILY_VOLUME: &gt;<br>                ${resources.volumes.content_historical.name}<br><br>              PENDING_FOLDER: &gt;<br>                ${var.ingestion_folder}<br>              PROCESSED_FOLDER: &gt;<br>                ${var.processed_folder}<br>              ERRORS_FOLDER: &gt;<br>                ${var.errors_folder} <br><br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_TOTALS_TABLE: &gt;<br>                ${var.bronze_profile_metrics_table}<br>              BRONZE_FOLLOWERS_TABLE: &gt;<br>                ${var.bronze_profile_followers_table}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              1. bronze ingestion/<br>              bronze historical ingest.ipynb<br>            source: WORKSPACE<br>        - task_key: trigger_transformation<br>          depends_on:<br>            - task_key: bronze_historical_ingest<br>          run_job_task:<br>            job_id: &gt;<br>              ${resources.jobs.linkedin_transformation.id}<br>      queue:<br>        enabled: true<br>      performance_target: PERFORMANCE_OPTIMIZED<br>    linkedin_post_ingest:<br>      name: linkedin post ingest<br>      trigger:<br>        pause_status: UNPAUSED<br>        file_arrival:<br>          url: &gt;<br>            /Volumes/${var.landing_catalog}/<br>            ${resources.schemas.landing.name}/<br>            ${resources.volumes.posts.name}/<br>            ${var.ingestion_folder}/<br>      tasks:<br>        - task_key: bronze_post_ingest<br>          notebook_task:<br>            base_parameters:<br>              LINKEDIN_PROFILE_NAME: &gt;<br>                ${var.linkedin_profile_name}<br><br>              LANDING_CATALOG: &gt;<br>                ${var.landing_catalog}<br>              LANDING_SCHEMA: &gt;<br>                ${resources.schemas.landing.name}<br>              LANDING_POSTS_VOLUME: &gt;<br>                ${resources.volumes.posts.name}<br>              <br>              PENDING_FOLDER: &gt;<br>                ${var.ingestion_folder}<br>              PROCESSED_FOLDER: &gt;<br>                ${var.processed_folder}<br>              ERRORS_FOLDER: &gt;<br>                ${var.errors_folder}<br><br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_IMPRESSIONS_TABLE: &gt;<br>                ${var.bronze_staging_impressions_prefix}<br>              BRONZE_POSTS_TABLE: &gt;<br>                ${var.bronze_post_metadata_table}<br>              BRONZE_POST_DETAILS_TABLE: &gt;<br>                ${var.bronze_post_metrics_table}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              1. bronze ingestion/<br>              bronze post ingest.ipynb<br>            source: WORKSPACE<br>        - task_key: trigger_transformation<br>          depends_on:<br>            - task_key: bronze_post_ingest<br>          run_job_task:<br>            job_id: &gt;<br>              ${resources.jobs.linkedin_transformation.id}<br>      queue:<br>        enabled: true<br>      performance_target: PERFORMANCE_OPTIMIZED<br>    linkedin_post_patch_ingest:<br>      name: linkedin post patch ingest<br>      trigger:<br>        pause_status: UNPAUSED<br>        file_arrival:<br>          url: &gt;<br>            /Volumes/${var.landing_catalog}/<br>            ${resources.schemas.landing.name}/<br>            ${resources.volumes.patch.name}/<br>            ${var.ingestion_folder}/<br>            ${var.patch_post_subfolder}/<br>      tasks:<br>        - task_key: bronze_post_patch_ingest<br>          notebook_task:<br>            base_parameters:<br>              LANDING_CATALOG: &gt;<br>                ${var.landing_catalog}<br>              LANDING_SCHEMA: &gt;<br>                ${resources.schemas.landing.name}<br>              LANDING_PATCH_VOLUME: &gt;<br>                ${resources.volumes.patch.name}<br>              <br>              PENDING_FOLDER: &gt;<br>                ${var.ingestion_folder}<br>              PROCESSED_FOLDER: &gt;<br>                ${var.processed_folder}<br>              ERRORS_FOLDER: &gt;<br>                ${var.errors_folder}<br>              PATCH_POST_SUBFOLDER: &gt;<br>                ${var.patch_post_subfolder}<br><br>              BRONZE_CATALOG: &gt;<br>                ${var.bronze_catalog}<br>              BRONZE_SCHEMA: &gt;<br>                ${resources.schemas.bronze.name}<br>              BRONZE_POST_PATCH_TABLE: &gt;<br>                ${var.bronze_post_patch_table}<br>            notebook_path: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              1. bronze ingestion/<br>              bronze post patch ingest.ipynb<br>            source: WORKSPACE<br>        - task_key: trigger_transformation<br>          depends_on:<br>            - task_key: bronze_post_patch_ingest<br>          run_job_task:<br>            job_id: &gt;<br>              ${resources.jobs.linkedin_transformation.id}<br>      queue:<br>        enabled: true<br>      performance_target: PERFORMANCE_OPTIMIZED</pre><p>There are a few changes to this set of Databricks Jobs that weren’t present in the POC pipeline:</p><ol><li>The post patch ingestion job now triggers on a file arrival event to the newly created patch landing volume, instead of on a bronze patch table update. This is because the bronze schema is being created by the DAB, and thus the bronze patch table can’t exist before that point, which makes the table update trigger invalid at the point of initial job creation. The post patch ingestion task has also been modified accordingly.</li><li>The gold materialized views have been moved into a Spark Declarative Pipeline; more on that later.</li><li>Task parameters have been defined for each notebook, bringing our defined constants into the notebooks.</li></ol><p>We also need to edit our notebooks to take in the task parameters. Here’s an example of what needs to be done for Python substitutions:</p><pre>import re<br><br>def get_valid_parameter(parameter_key: str):<br>    &quot;&quot;&quot;<br>    Get a valid parameter value from the Databricks widgets.<br>    Hardened to thwart SQL injection attacks.<br>    &quot;&quot;&quot;<br>    parameter_value = dbutils.widgets.get(parameter_key)<br>    <br>    # Parameter_value must be a string with only <br>    # alphanumeric characters and underscores<br>    if not re.fullmatch(r&#39;[a-zA-Z0-9_]+&#39;, parameter_value):<br>        raise ValueError(<br>            f&quot;Invalid parameter value for {parameter_key}: &quot;<br>            f&quot;{parameter_value}&quot;<br>        )<br>    <br>    # Disallow dangerous SQL keywords and patterns<br>    forbidden_patterns = [<br>        r&#39;--&#39;, r&#39;;&#39;, r&quot;&#39;&quot;, r&#39;&quot;&#39;, r&#39;/\*&#39;, r&#39;\*/&#39;, r&#39;xp_&#39;, r&#39;char\(&#39;, <br>        r&#39;nchar\(&#39;, r&#39;varchar\(&#39;, r&#39;\balter\b&#39;, r&#39;\bdrop\b&#39;, <br>        r&#39;\binsert\b&#39;, r&#39;\bdelete\b&#39;, r&#39;\bupdate\b&#39;, <br>        r&#39;\bselect\b&#39;, r&#39;\bcreate\b&#39;, r&#39;\bexec\b&#39;, r&#39;\bunion\b&#39;, <br>        r&#39;\bor\b&#39;, r&#39;\band\b&#39;<br>    ]<br>    for pattern in forbidden_patterns:<br>        if re.search(pattern, parameter_value, re.IGNORECASE):<br>            raise ValueError(<br>              f&quot;Potentially dangerous value for {parameter_key}: &quot;<br>              f&quot;{parameter_value} (pattern matched: {pattern})&quot;<br>            )<br>    return parameter_value<br><br># 1. define our constants<br>try:<br>    BRONZE_CATALOG = get_valid_parameter(&quot;BRONZE_CATALOG&quot;)<br>    BRONZE_SCHEMA = get_valid_parameter(&quot;BRONZE_SCHEMA&quot;)<br>    BRONZE_IMPRESSIONS_TABLE_PREFIX = \<br>      get_valid_parameter(&quot;BRONZE_IMPRESSIONS_TABLE_PREFIX&quot;)<br>    BRONZE_TOTALS_TABLE = get_valid_parameter(&quot;BRONZE_TOTALS_TABLE&quot;)<br><br>    SILVER_CATALOG = get_valid_parameter(&quot;SILVER_CATALOG&quot;)<br>    SILVER_SCHEMA = get_valid_parameter(&quot;SILVER_SCHEMA&quot;)<br>    SILVER_IMPRESSIONS_TABLE = \<br>      get_valid_parameter(&quot;SILVER_IMPRESSIONS_TABLE&quot;)<br><br>    print(&quot;Loaded widget values&quot;)<br>except:<br>    BRONZE_CATALOG = &quot;bronze&quot;<br>    BRONZE_SCHEMA = &quot;linkedin&quot;<br>    BRONZE_IMPRESSIONS_TABLE_PREFIX = &quot;impressions&quot;<br>    BRONZE_TOTALS_TABLE = &quot;totals&quot;<br><br>    SILVER_CATALOG = &quot;silver&quot;<br>    SILVER_SCHEMA = &quot;linkedin&quot;<br>    SILVER_IMPRESSIONS_TABLE = &quot;impressions&quot;<br><br>    print(&quot;Failed to load widget values, using default values&quot;)</pre><p>We could have retrieved the parameters directly using Databricks’ associated utility function, but instead we’re wrapping it in additional Python code to prevent <a href="https://www.w3schools.com/sql/sql_injection.asp">SQL injection attacks</a> (at least on a basic level).</p><p>SQL code is a tougher ask. In theory, Databricks recommends the use of the IDENTIFIER clause for bringing parameters into SQL code, as per below:</p><pre>CREATE VIEW IF NOT EXISTS IDENTIFIER <br>  (:SILVER_CATALOG || &quot;.&quot; || :SILVER_SCHEMA || &quot;.&quot; || <br>   :SILVER_TOTALS_TABLE) <br>AS <br>  SELECT date, impressions, engagements <br>FROM IDENTIFIER <br>  (:BRONZE_CATALOG || &quot;.&quot; || :BRONZE_SCHEMA || &quot;.&quot; || <br>   :BRONZE_TOTALS_TABLE);<br><br><br>CREATE VIEW IF NOT EXISTS IDENTIFIER <br>  (:SILVER_CATALOG || &quot;.&quot; || :SILVER_SCHEMA || &quot;.&quot; || <br>   :SILVER_FOLLOWERS_TABLE) <br>AS <br>  SELECT date, new_followers <br>FROM IDENTIFIER <br>  (:BRONZE_CATALOG || &quot;.&quot; || :BRONZE_SCHEMA || &quot;.&quot; || <br>   :BRONZE_FOLLOWERS_TABLE);</pre><p>But the IDENTIFIER clause, as of this writing, doesn’t support DDL statements like the above, which unfortunately covers all instances of our parameter usage. According to <a href="https://community.databricks.com/t5/data-engineering/identifier-in-sql-views-not-supported/td-p/141409">a Databricks forum thread</a>, this issue should go away once the Databricks runtime on serverless computes are upgraded to the latest version, so this is hopefully a temporary state of affairs.</p><p>We can use the Python approach to validate our parameters before executing the SQL code using the <strong>spark.sql</strong> function. This is what we end up doing for our logical view creation in the silver layer.</p><p>But there is an additional wrinkle at the gold layer: SQL statements for materialized views can only run on an SQL warehouse cluster. This precludes using Python to retrieve the parameters properly.</p><p>This is why the code for gold materialized views have been moved into a pipeline that replaces the old notebook: that is what we’ll look at next.</p><h4>e. pipelines.yml</h4><p>First, here’s the pipeline definition for reference:</p><pre>resources:<br>  pipelines:<br>    gold_create:<br>      name: gold_create<br>      configuration:<br>        SILVER_CATALOG: &gt;<br>          ${var.silver_catalog}<br>        SILVER_SCHEMA: &gt;<br>          ${resources.schemas.silver.name}<br>        SILVER_IMPRESSIONS_TABLE: &gt;<br>          ${var.silver_post_impressions}<br>        SILVER_ENGAGEMENTS_TABLE: &gt;<br>          ${var.silver_post_engagements}<br>        SILVER_POSTS_TABLE: &gt;<br>          ${var.silver_post_metadata}<br>        SILVER_TOTALS_TABLE: &gt;<br>          ${var.silver_profile_metrics}<br>        SILVER_FOLLOWERS_TABLE: &gt;<br>          ${var.silver_profile_followers}<br>          <br>        GOLD_CATALOG: &gt;<br>          ${var.gold_catalog}<br>        GOLD_SCHEMA: &gt;<br>          ${resources.schemas.gold.name}<br>        FCT_DAILY_PROFILE_STATS_TABLE: &gt;<br>          ${var.gold_fact_daily_profile_statistics_table}<br>        FCT_DAILY_POST_STATS_TABLE: &gt;<br>          ${var.gold_fact_daily_post_statistics_table}<br>        DIM_DATE_TABLE: &gt;<br>          ${var.gold_dimension_date_table}<br>        DIM_TIMESTAMP_TABLE: &gt;<br>          ${var.gold_dimension_timestamp_table}<br>      libraries:<br>        - glob:<br>            include: &gt;<br>              ../src/linkedin_analytics_jobs/<br>              3. gold modelling/pipeline_gold_create/**<br>      schema: ${resources.schemas.gold.name}<br>      continuous: false<br>      development: false<br>      photon: true<br>      channel: CURRENT<br>      catalog: ${var.gold_catalog}<br>      serverless: true<br>      root_path: &gt;<br>        ../src/linkedin_analytics_jobs/<br>        3. gold modelling/</pre><p>Just like with the Databricks Jobs, I created a POC manually and configured it to work, then copied the YAML over to the DAB and modified it accordingly. I’ve also copied the associated pipeline files to the DAB, resulting in the following file structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*a_uanLJHxsdrk1J1todBMw.png" /><figcaption>Structure of the gold modelling folder in the DAB. Note the icon next to the three dots; clicking that brings us to the associated pipeline editor. This icon only appears after you have deployed the DAB at least once.</figcaption></figure><p>Here are the highlights:</p><ol><li>Each gold materialized view definition has been separated into its own SQL file in the same folder (<strong>3. gold modelling/pipeline_gold_create</strong>).</li><li>For now, each gold materialized view uses <a href="https://docs.databricks.com/aws/en/ldp/parameters?language=SQL">the older <strong>${PARAM}</strong> parameter syntax</a> because IDENTIFIER does not yet support our DDL materialized view statements on serverless SQL warehouses. Once the runtime is upgraded, these scripts should be migrated to parameter markers plus IDENTIFIER.</li></ol><p>See the <a href="https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-names-identifier-clause?language=SQL">documentation for IDENTIFIER</a> for more details.</p><blockquote>Deprecation of language features, syntax and other dependencies can be a real concern in a production environment. The general guideline is to avoid using deprecated features and to migrate pipelines that depend on such features in a timely fashion and wherever feasible.</blockquote><p>One last note before we move on: the risk arising from an SQL injection attack in this instance (a personal LinkedIn project) is relatively low. In addition, our scripts are internal and parameters are controlled values coming from the DAB, which greatly reduces the likelihood of an SQL injection attack.</p><h4>f. <strong>databricks.yml</strong></h4><p>Now for the main YAML file.</p><pre># This is a Databricks asset bundle definition for &lt;DAB name&gt;.<br># See https://docs.databricks.com/dev-tools/bundles/index.html <br># for documentation.<br>bundle:<br>  name: &lt;DAB name&gt;<br>  uuid: &lt;DAB UUID&gt;<br><br>include:<br>  - resources/*.yml<br>  - resources/*/*.yml<br>targets:<br>  dev:<br>    # The default target uses &#39;mode: development&#39; <br>    # to create a development copy.<br>    # - Deployed resources get prefixed with &#39;[dev my_user_name]&#39;<br>    # - Any job schedules and triggers are paused by default.<br>    # See also <br>    # https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.<br>    mode: development<br>    default: true<br>    workspace:<br>      host: https://&lt;redacted&gt;.cloud.databricks.com<br>  prod:<br>    mode: production<br>    workspace:<br>      host: https://&lt;redacted&gt;.cloud.databricks.com<br>      # We explicitly deploy to /Workspace/Users/&lt;current user&gt; <br>      # to make sure we only have a single copy in the workspace.<br>      root_path: &gt;<br>        /Workspace/Users/<br>        ${workspace.current_user.userName}/<br>        .bundle/${bundle.name}/${bundle.target}<br>    run_as:<br>      service_principal_name: &gt;<br>        ${var.pipeline_runner}<br>    variables:<br>      dashboard_subscriber: &gt;<br>        ${workspace.current_user.userName}<br>    # permissions:<br>    #   - user_name: &lt;replace with your own user name&gt;<br>    #     level: CAN_MANAGE</pre><p>Here we see the service principal being used to run the production pipelines, and the other YAML files being included via the defined paths.</p><p>We are working on the assumption that we have only one workspace in one Databricks instance to work with, and indeed Databricks Free Edition only allows a single workspace per instance.</p><p>In an enterprise deployment, development and production environments would typically be deployed in different workspaces or different instances; the latter applies where the Databricks instances are associated with a development and production cloud project (e.g. a Google Cloud project or an Azure resource group).</p><h3>V. Deploying the DAB</h3><p>Clicking the rocket icon below the folders icon in the DAB interface brings up the Deployments navigation panel. Click ‘Deploy’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/1*ZhrzIIyQLmybEuCUeFRpkA.png" /><figcaption>A portion of the DAB interface, showing the ‘Deploy’ button.</figcaption></figure><p>You will be presented with a deployment screen with a progress wheel; this is where Databricks is running in the background to check what can be deployed (equivalent to the ‘terraform plan’ step). If this step fails, there is most likely something wrong with your DAB configuration, and you will need to troubleshoot accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*OTonzv6PTDTvL-9gaKbSuA.png" /><figcaption>Deployment screen with changes being reviewed.</figcaption></figure><p>If the check is successful, you will be shown a list of all resources that are to be created, deleted or updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*nJxMdAIYJkA5qYhWB7dQnQ.png" /><figcaption>Deployment screen with all changes loaded.</figcaption></figure><p>Note the dev prefixes for the dashboards, jobs, pipelines and schemas; this is due to the <strong>‘development: true’</strong> setting in databricks.yml, and is a way to separate the development deployments being created by multiple contributors in the same workspace.</p><p>Clicking the ‘Deploy’ button will bring up the deployment output, where you can track the progress of the deployment (this is the equivalent of ‘terraform apply’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*rmGCYnqe5P1DWUVegxnC0w.png" /><figcaption>Example log of a successful deployment.</figcaption></figure><p>If the deployment is successful, you will see the list of resources created as below. Now you can go to the respective pipelines and test them accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*63jvC1P05heU1OWUMbwAzg.png" /><figcaption>Example list of deployed DAB development resources.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*gdloLnH-vPynTMr0QgUIqw.png" /><figcaption>Example of a landing schema and volumes created by a DAB.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*-s-tCYTxGv6t-bRtmNRC6Q.png" /><figcaption>Example list of Databricks Jobs created by a DAB.</figcaption></figure><p>Once we’re happy with the results, we can deploy the DAB to production, and test there again.</p><p>Below is a list of additional fixes I did to the pipeline that were not immediately apparent in the POC:</p><ol><li>I added statements where relevant to create empty source tables, particularly for <strong>bronze.linkedin.post_details</strong> (which only populates when post analytics Excel files are ingested)</li></ol><h3>V. Areas of Improvement</h3><p>We now have a maintainable pipeline where we can experiment in a development environment, with the confidence that we can implement or roll back any changes rapidly and accurately.</p><p>There are a few areas that we have not covered that are beyond the scope of the above deployment.</p><ol><li>The ‘pending’ folders in each volume still need to be created manually; Databricks cannot define volume folders directly in asset bundles as of this writing. This can be mitigated by including a small Databricks Job that runs a script or notebook to create said folders.</li><li>We have been working exclusively in the Databricks UI, which is fine for our demonstration purposes. In an enterprise deployment we should be working off of the associated Git repository and following the relevant best practices, including raising pull requests for feature changes and committing them to separate development and production (main) branches. For this to work, we will need to configure the relevant CI/CD actions (e.g. GitHub Actions on GitHub, or Jenkins for an agnostic orchestrator) to make use of the Databricks CLI to manage the DAB. This is also where the ‘pending’ folders can be created, and the queries in our Databricks dashboard JSON updated, as part of the follow-up CI/CD actions. The end result is to make deployments reproducible and reviewable, rather than dependent on manual clicks in the UI.</li></ol><h3>VI. What’s Next?</h3><p>Once the above deployment gaps are closed, the final step is to know when the pipeline breaks, and to see at a glance where the pipeline may be breaking. That’s the subject of our next post. See you there!</p><p>Yingzhao Ouyang is an AI and data engineering specialist with a distinctive blend of humanities, business, and technical expertise, bringing a uniquely holistic perspective to enterprise data challenges that others with purely technical backgrounds miss. To find out more, visit his website at <a href="https://www.yzouyang.com/">https://www.yzouyang.com/</a> or follow his LinkedIn profile at <a href="https://www.linkedin.com/in/yzouyang/">https://www.linkedin.com/in/yzouyang/</a></p><p>This post is also available on Yingzhao Ouyang’s website: <a href="https://www.yzouyang.com/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline/">https://www.yzouyang.com/build-your-own-linkedin-analytics-part-9-making-a-maintainable-pipeline/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2c35a874e3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>