<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by jacob baruch on Medium]]></title>
        <description><![CDATA[Stories by jacob baruch on Medium]]></description>
        <link>https://medium.com/@baruchjacob?source=rss-ba92e1017bc3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZyWQjrqI3PNsi31HTDDp9g.jpeg</url>
            <title>Stories by jacob baruch on Medium</title>
            <link>https://medium.com/@baruchjacob?source=rss-ba92e1017bc3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:44:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@baruchjacob/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Minsky’s frames, 1974: the idea your data agent is missing]]></title>
            <link>https://medium.com/@baruchjacob/minskys-frames-1974-the-idea-your-data-agent-is-missing-23a14579bef5?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/23a14579bef5</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Fri, 20 Nov 2026 00:00:28 GMT</pubDate>
            <atom:updated>2026-11-20T09:55:44.176Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*38UirD--1GYZalk9JD_zGg.png" /><figcaption>After Minsky, 1974: a frame, its defaults, the exceptions it can explain, and the row it cannot. Illustration, not measured data.</figcaption></figure><p>TL;DR. In 1974 Marvin Minsky argued that people understand a new situation by pulling a remembered structure, which he called a frame, and adjusting the details. A frame has slots, a default for each slot, and room for known exceptions. An AI agent reading your tables has a generic frame from its training, not yours. It sees column names and guesses what a row means. Write one short frame card per important table, and tell the agent to stop and ask when a row fits nothing on it.</p><h3>What did Minsky propose?</h3><p>Marvin Minsky <a href="https://amturing.acm.org/award_recipient/minsky_7440781">co-founded the AI lab at MIT</a>. In June 1974 he wrote a memo called <a href="https://dspace.mit.edu/handle/1721.1/6089">A Framework for Representing Knowledge</a>. In a <a href="https://www.youtube.com/watch?v=mKtEIl-D_wY">two-minute interview clip from 2011</a> he explains the name himself. His starting point was that people do not reason from scratch. When you open the door to a room you have never seen, you already expect walls, a floor and a ceiling, and you notice only what breaks the pattern.</p><p>He called that remembered structure a frame. A frame has a top part that is always true, and slots that get filled for each case. Each slot has conditions on what may fill it, and a default used until something better arrives. When something does not fit, you can find an excuse: a chair that is too small still fits the chair frame if it is a toy, so you keep the frame and note the exception. When nothing explains the mismatch, you switch frames. Drawing on a colleague’s work, he added that checking effort scales with the stakes: a desk gets a glance, a unicorn gets a close look. He also called a frame “a collection of questions to be asked” about a situation, and said plainly that the theory was incomplete.</p><h3>Why does a 1974 idea matter now?</h3><p>Frames went on to become a standard part of the <a href="https://amturing.acm.org/award_recipient/minsky_7440781">expert-system software of the 1980s</a>. I think those systems faded because every frame had to be written by hand, and language models now learn their frames from text instead.</p><p>The problem frames solved did not go away. An agent that answers questions about your data reads tables the way a new hire reads them on day one: it knows the words, not your business. Industry analysts at Forrester <a href="https://www.forrester.com/blogs/build-meaning-before-machines-why-semantics-ontologies-and-knowledge-graphs-matter-for-agentic-ai/">warned in 2026</a> that agents without explicit context get joins wrong and misread metrics. AI engineers describe <a href="https://latent.space/p/ontologies-agentic-systems/">rediscovering ontologies</a> to fill the gap, and much of that writing comes from people who sell the machinery.</p><p>I think a startup does not need an ontology. It needs the frame a good analyst carries in their head, written down for the few tables your key numbers come from.</p><h3>What does a frame look like for a data table?</h3><p>Take an orders table. A person who knows the business reads it with a frame already loaded. One row is an order attempt by a real customer, and only paid ones count as revenue. The amount is above zero and in dollars. Test accounts, refunds and orders not yet paid are known exceptions. Revenue deserves a close look, because the board sees it.</p><p>The agent has none of this unless someone writes it down. Picture a startup where the engineers test checkout with company email addresses. The agent sums the amount column, the test orders are inside, and the founder quotes a revenue figure about four times too high, as in the test below.</p><h3>A small test: does a frame card change the answer?</h3><p>I ran this on 20 November 2026 with eight made-up orders and one question: what was our revenue in October? The correct answer, computed in SQL, is 320 dollars, plus one euro order that a person should look at. I asked an open model from OpenAI, gpt-oss-120b on Cloudflare Workers AI, three times with the table alone and three times with a frame card above it.</p><p>Without the card, all three answers counted the two test accounts and came to 1,369 or 1,449 dollars, more than four times too high. With the card, two answers came to 320 dollars and listed what was left out and why. The third stopped and asked what to do with the euro order, which is what the card told it to do. A smaller model, Llama 4 Scout, also left the test accounts out once it had the card, but it got the sums wrong and never stopped to ask.</p><p>Three runs per case is a direction, not a measurement. It shows two things: the card changes what gets counted, and the sums should run in SQL rather than in the model’s head.</p><h3>What goes on the card?</h3><p>Each part of the card answers one failure.</p><p><strong>The default meaning.</strong> Failure: the agent counts what a row looks like, not what it is. Guardrail: one sentence at the top of the card says what one row is. Check: ask the agent what one row in the table is, and compare.</p><p><strong>The known exceptions.</strong> Failure: test accounts and refunds go into customer numbers. Guardrail: list each exception and what to do with it. Check: run the query below and look at the exception counts.</p><p><strong>When to stop.</strong> Failure: a row that fits nothing is dropped or guessed. Guardrail: a rule that such rows go to a person, and a query that counts them. Check: the count of rows that fit no slot is reported with every answer.</p><p><strong>How much checking.</strong> Failure: a board number gets the same care as a page view count. Guardrail: name the high-stakes numbers on the card. Check: for those numbers the agent shows its query.</p><h3>What changes by stage?</h3><p>Whether an agent answers data questions at all matters more here than stage. With no agent, a person carries the frame and this can wait.</p><p><strong>Before launch.</strong> You have one or two tables and no agent asking questions yet. Write the default meaning as a comment on the table and stop.</p><p><strong>First paying customers.</strong> Write cards for the tables your revenue and active-customer numbers come from, usually orders, customers and subscriptions.</p><p><strong>Growing, with a first data person.</strong> That person owns the cards, adds exceptions as they are found, and turns the stop rule into a check that runs every day.</p><p>For your data team</p><pre># orders: frame card One row is: one order attempt by a customer. Revenue counts only paid orders. Default: status paid, amount positive, currency USD, a real customer. Known exceptions (keep the frame, leave the row out of revenue or net it): - Emails ending @ourco.com are our own test and internal accounts. Leave them out of every customer number. - status refunded: a refund of an earlier order, amount is negative. Subtract it from revenue in the month it happens. - status pending: not paid yet. Not revenue. Stop and ask a person when a row fits none of the above, for example a currency other than USD. Do not convert or guess. Stakes: revenue is a board number. State what you excluded and why.</pre><pre>create view orders_frame as select o.*, case when lower(customer_email) like &#39;%@ourco.com&#39; then &#39;exception: internal or test account&#39; when currency is distinct from &#39;USD&#39; then &#39;no slot fits: currency&#39; when status = &#39;refunded&#39; then &#39;exception: refund&#39; when status = &#39;pending&#39; then &#39;exception: not paid yet&#39; when status = &#39;paid&#39; and amount &gt; 0 then &#39;fits&#39; else &#39;no slot fits: other&#39; end as frame_match from orders o; select frame_match, count(*), sum(amount) from orders_frame group by 1 order by 1;</pre><h3>Check it in 10 minutes</h3><ol><li>Ask your agent what one row in your orders table is. If the answer is vague, write the one-sentence default meaning.</li><li>Ask how many orders came from your own team’s email addresses this month. If the number is not zero, add that exception to the card.</li><li>Ask for last month’s revenue and how it was counted. If the agent added the numbers itself, tell it to run SQL.</li><li>No coding agent and no repository? This step is for you. Paste the card above into the instructions of the AI tool your team uses for data questions, with your own email domain and statuses, and ask the same question again.</li><li>The person who answers for revenue in board meetings owns the card.</li></ol><p>See where your setup stands with the <a href="https://goagentdata.com/#self-check">two minute self-check</a>. The <a href="https://goagentdata.com/posts/day-one-metric">day-one metric post</a> shows how test accounts distort the first number you track.</p><p><em>Originally published at </em><a href="https://goagentdata.com/posts/minsky-frames-for-data-agents"><em>https://goagentdata.com</em></a><em> on November 20, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23a14579bef5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your agent has an opinion. You may think it’s yours]]></title>
            <link>https://medium.com/@baruchjacob/your-agent-has-an-opinion-you-may-think-its-yours-3b90a03f3ae6?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b90a03f3ae6</guid>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 19:25:29 GMT</pubDate>
            <atom:updated>2026-11-03T19:25:29.089Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR.</strong> An AI agent can steer a data decision without anyone noticing. In a study Anthropic published in October 2026, 50 analysts matched an agent’s secretly assigned lean 86 percent of the time, and few credited it. To keep the decision yours, write your measure of success and your expected result before you open the agent.</p><p>Checking the number rarely catches this, because a judgment call has no single right answer. So ask the agent to list its choices, and rerun the ones that could change the decision.</p><figure><img alt="Two flows side by side. Agent first: the agent’s lean moves your view unnoticed, and the recommendation carries that lean. You first: a sealed note holds your measure and expected result, you compare it with the agent’s answer, rerun with the other option, and a named person decides." src="https://cdn-images-1.medium.com/max/1024/1*Wgy9zjBmSTG783tjH7K6dg.jpeg" /><figcaption>Illustration of the method, not measured data.</figcaption></figure><h3>Do AI agents sway data decisions?</h3><p>Picture a founder asking an agent whether the new onboarding worked. The agent must choose <a href="https://goagentdata.com/posts/one-definition-per-metric">what counts as success</a>, which customers to compare, and whether one huge account stays in. Together, those choices can decide the answer.</p><p>Anthropic, the company that makes the model tested, <a href="https://anthropic.com/research/agents-judgment-calls">ran an experiment on this</a> and published it on 27 October 2026. Analysts advised a fictional farming cooperative with an AI coding agent. For each of four analysis decisions, a coin flip set which option the agent’s hidden instructions told it to favor.</p><ul><li><strong>They followed it.</strong> Choices matched the agent’s lean 86 percent of the time, where chance would be about 50.</li><li><strong>It changed minds.</strong> In 38 cases an analyst took a side first. They switched 60 percent of the time when the lean went against them and 6 percent when it agreed, an imprecise estimate (the gap ranges from 24 to 82 points).</li><li><strong>The final call moved less clearly.</strong> 54 percent recommended rollout when the agent leaned that way, and 36 percent otherwise. On the paper’s recommendation score, the adjusted range spans minus 6 to 40 points, so it could be zero.</li><li><strong>They did not see it.</strong> 77 percent repeated the agent’s arguments in at least one answer, and only 15 percent credited it anywhere. None suspected the agent’s preferences had been manipulated.</li></ul><p>It is one task and one model, and the researchers deliberately strengthened the lean, so everyday effects may be smaller. They also write that this is no argument for using AI less. With no lean assigned, the same model made the same five choices in 112 of 123 runs, so a default can act like a lean.</p><h3>Haven’t we seen this in meetings?</h3><p>Partly, I think. The first-speaker link is my reading. The authors tie their result to automation bias, over-relying on automated tools.</p><p>Before agents, an analyst made these choices and a colleague pushed back. In a <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC3107299">2011 experiment</a>, 144 students estimated facts, some seeing what others had answered. Their estimates grew more alike without becoming more accurate, and their confidence rose anyway. Anchoring is the other half: <a href="https://sites.socsci.uci.edu/~bskyrms/bio/readings/tversky_k_heuristics_biases.pdf">a first number pulls later guesses toward it</a>, as Tversky and Kahneman showed in 1974. I think that is why writing answers down first helps.</p><p>An agent is that first voice, with three differences:</p><ul><li><strong>It often goes first.</strong> Told to raise its lean, the agent <a href="https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf">voiced a side first</a> on 77 percent of 245 decisions, the analyst on 17.</li><li><strong>Nobody else is in the room.</strong> A meeting has a second person who might object.</li><li><strong>It argues its case.</strong> Even analysts who spoke first often switched, and its reasons can feel like your own.</li></ul><h3>Why doesn’t checking the answer catch it?</h3><p>The <a href="https://goagentdata.com/posts/evals-for-data-answers">evals in our earlier post</a> catch a wrong number. They cannot tell you whether seven-day activation beats 30-day retention, because that depends on your business.</p><p>So “a person reviews it” is a weak control here, because the reviewer reads the answer first. Ask reviewers for one line of what they expect before they open it.</p><h3>How do you stop an agent steering your analysis?</h3><p><strong>Write first.</strong> Before you open the agent, write one short note: the decision, the measure of success, and the result you expect. You need this only for decisions about product, price, spend or hiring. If your metric is not defined in writing yet, that is the real work.</p><p>In a <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC10772030/">2024 study</a>, people judged mock defendants with an AI that was wrong once in three. In the first experiment, with 150 laypeople, 66 percent of those who judged before seeing the AI were right on the case where it erred, against 37 percent who had seen it first. But once everyone had seen the wrong advice, accuracy was not significantly different.</p><p>Writing first protects your view, not your reaction to the reply, so you need a second step.</p><p><strong>Compare, then rerun.</strong> Where your note and the answer differ, do not reread the agent’s argument. Rerun with your option and see whether the conclusion moves.</p><p><strong>Ask for the choices.</strong> You need no repository for this. Paste the prompt below with your question, and leave your guess out of it, because the assistant may simply agree with it.</p><pre>The measure that decides this is: &lt;your measure&gt;.<br>Before you recommend anything:<br>1. List the choices you made: what you measured, who you counted,<br>   the dates, and what you left out.<br>2. For each, say what the other sensible option was.<br>3. Run it again with the other option and show both results side by side.<br>4. Tell me which choices change the answer.<br>Then give your recommendation in plain words.</pre><p>In a test on a made-up data set where the new onboarding raised activation but not paid conversion, the assistant recommended keeping it when no measure was named, even with this prompt. With the measure named, it said the data showed no win. One run each with one model, so this illustrates, it does not measure.</p><h3>Can’t we just think harder?</h3><p>Only partly. In the study, experience, expertise, and measures of open-minded thinking and enjoying hard thinking did not predict who followed the agent, though 50 people is a small test.</p><p>Training helps modestly. In a <a href="https://www.sciencedirect.com/science/article/pii/S036013152600014X">2026 study of 226 students</a>, a 2-hour session combining a questioning AI tutor with role play reduced how much groups copied from AI, without raising self-rated critical thinking. A <a href="https://arxiv.org/abs/2503.10556">short reading on how ChatGPT works</a> did not significantly reduce following wrong suggestions among 36 high school students in a 2025 preprint.</p><p>Written prompts show mixed results. In a <a href="https://arxiv.org/html/2601.18033v1">2026 preprint with 214 people</a> on AI-assisted writing, people who named the assumption behind an AI’s plan accepted 63 percent of flawed outputs, against 71 with no prompt, a gap that was not significant. A what-if prompt scored 81, and combining the two did not help. I still think of the note and the prompt as critical thinking on paper, but the evidence is thin.</p><h3>What does this look like at each stage?</h3><p>I think stakes matter more than stage: a pricing call before launch deserves more care than a button test later. Stage decides who does the work.</p><p><strong>Before launch.</strong> Data is thin, so this mostly applies to pricing and market questions. Write the note in a chat message and use the prompt above.</p><p><strong>First paying customers.</strong> Name one owner per decision, and keep dated notes in a shared doc. Whoever asked the agent runs the prompt.</p><p><strong>Growing with a first data person.</strong> The data person sets which decisions need a note, reviews the choices as well as the numbers, and moves the notes and the rule into the repository.</p><h3>What goes wrong, and how do you catch it?</h3><p><strong>The agent chose the measure.</strong> You asked whether onboarding worked, and it <a href="https://goagentdata.com/posts/ai-needs-data-and-context">filled the gap</a> with seven-day activation.<br>Guardrail: the decision owner names the measure in the note before the agent runs.<br>Check: the decision owner compares the note’s date with the analysis date. The note must be older.</p><p><strong>The conclusion rests on one quiet choice.</strong> Dropping one large account flips the result, and nobody mentions it.<br>Guardrail: the person who ran the analysis shows both results in one table.<br>Check: the decision owner asks the agent to rerun with the account kept in. If the result crosses the line in the note, the choice is the decision.</p><h3>For your data team</h3><p>This part gets technical, and it assumes a repository and a coding agent. Without those, the note and the prompt above are enough. The standard techniques are a pre-analysis plan (preregistration), a sensitivity check that changes one choice at a time, and, with many defensible choices, <a href="https://www.nature.com/articles/s41562-020-0912-z">specification curve analysis</a>.</p><p>The note, committed before the session:</p><pre># decisions/2026-11-onboarding.yaml<br>decision: keep or roll back the new onboarding<br>owner: &lt;named person&gt;<br>written_on: 2026-11-04<br>success_measure: paid conversion within 30 days of sign-up<br>who_is_counted: new accounts, excludes is_test and is_internal<br>window: sign-ups from &lt;start&gt; to &lt;end&gt;, only cohorts at least 30 days old<br>comparison: &lt;new vs old onboarding, and how accounts were assigned&gt;<br>changes_the_decision_if: &lt;the lift at which you would keep it&gt;</pre><p>Your expected result stays out of this file, since the agent reads it.</p><p>A rule for your agent’s instructions file, such as CLAUDE.md, or AGENTS.md for Cursor. It is part of the <a href="https://goagentdata.com/posts/data-agent-harness">harness</a>:</p><pre>For any analysis that feeds a decision about product, price, spend or hiring<br>(skip plain lookups such as counts and status checks):<br>1. Read the matching note in decisions/. If none exists, ask for one first.<br>2. Open with the judgment calls (measure, who is counted, window, exclusions,<br>   method), the option you used for each, and one line on the case for the<br>   main alternative. Then give your recommendation and name the calls it<br>   depends on. If asked for one line, give it and name the call that would<br>   change it.<br>3. For each call that could reasonably go another way, rerun with the other<br>   option, one change at a time, and show the results in one table. Say which<br>   calls flip the conclusion.<br>4. Use read-only data access. Write only to analysis/, never to production.</pre><p>Caveats:</p><ul><li>Whether rule 2 lowers deference is untested.</li><li>The rule guides the agent. Read-only credentials enforce only the write limit.</li></ul><h3>Check it in 10 minutes</h3><p>Take the last agent analysis that fed a decision.</p><ol><li>Ask the agent which filters, dates, thresholds and definitions could reasonably have gone another way.</li><li>Mark each one: did a person decide this before seeing the result?</li><li>Pick the unmarked choice that touches the headline number. Rerun a copy with the other option, same data.</li><li>If the conclusion flips, the answer depends on a choice nobody made on purpose. Name who owns it. A flip does not say which option is right.</li></ol><p>See where your setup stands with the <a href="https://goagentdata.com/#self-check">two minute self-check</a>. Then read the <a href="https://goagentdata.com/guide#checks">draft checks</a> your coding agent can run.</p><p>Originally published at <a href="https://goagentdata.com/posts/your-agent-has-an-opinion">goagentdata.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b90a03f3ae6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI makes data answers cheap. Judging them is the new data work]]></title>
            <link>https://medium.com/@baruchjacob/ai-makes-data-answers-cheap-judging-them-is-the-new-data-work-19fa79d24692?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/19fa79d24692</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 00:00:07 GMT</pubDate>
            <atom:updated>2026-10-26T18:41:13.413Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR.</strong> When an AI assistant makes a data answer almost free, people ask far more questions, and the scarce part moves from writing the query to judging the answer. AI does remove real work, such as routine queries and one-off reports. It also adds work the model cannot do for you: deciding what your business means by “customer”, which data to trust, and whether an answer is right.</p><p>Who owns that judgment, and when, depends on how many people ask, how much money depends on the answers, and how many of your definitions are contested.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ub_DaqL7nBnCjsArF9Clpw.png" /></figure><h3>What happens when an answer gets cheap?</h3><p>Before an AI assistant, a data question had a price. You wrote a ticket, waited for the analyst, and often decided the question was not worth it. In my experience, many questions were never asked.</p><p>Here is how I think about it. An assistant drops that price close to zero. A head of sales can ask which region is losing deals at eleven at night and get a clear answer in seconds. So she asks more, and so does everyone else. It is not only people. Other agents and scheduled dashboards now send questions to the same assistant, and nobody reads each answer.</p><p>Economists call this pattern the <a href="https://en.wikipedia.org/wiki/Jevons_paradox">Jevons paradox</a>. In 1865, W. S. Jevons argued in The Coal Question that more efficient use of coal led Britain to burn more coal, not less. When something gets cheaper to use, people often use much more of it. Data answers may follow the same pattern. That is an argument, not a measurement.</p><p>The spreadsheet is the closest example. After it arrived, <a href="https://www.npr.org/sections/money/2015/02/25/389027988/how-the-electronic-spreadsheet-revolutionized-business">NPR’s Planet Money reported</a> that bookkeeping and accounting clerk jobs fell while accountant jobs grew. The report gives no numbers. I read it this way: the routine work shrank, and the work of deciding what the numbers mean grew.</p><h3>What does AI actually remove?</h3><p>I think it removes a lot, and pretending otherwise makes the rest of this argument weaker.</p><ul><li><strong>Routine queries.</strong> “How many sign-ups last week” no longer needs a person who writes SQL.</li><li><strong>One-off reports.</strong> The spreadsheet someone builds for a single meeting and never opens again.</li><li><strong>Simple lookups.</strong> The ticket queue that kept a junior analyst busy.</li><li><strong>First drafts of analysis.</strong> A first cut at why a number moved, which a person then checks.</li></ul><p>It also helps the data person: it can draft the definitions file and the checks described below.</p><p>If your company has one product, one billing system and a founder who asks a few questions a week, this may be most of your data work. For you, AI can mean you need less data help, and for longer.</p><h3>What does it add?</h3><p>Here is the part that does not get cheaper. Each item is a decision about your business, not a technical task.</p><ul><li><strong>Definitions.</strong> Does a customer on a free trial count as a customer? Does a refunded sale count as revenue? The model can find the table, but it cannot know an answer your company never decided.</li><li><strong>Trusted sources.</strong> When billing and your CRM (the tool where sales tracks customers) disagree on revenue, someone has to decide which one is the trusted number.</li><li><strong>Checking answers.</strong> The head of sales cannot read the query behind her answer. A wrong answer looks exactly like a right one, so she forwards it to whoever owns the data and asks whether it is right.</li><li><strong>Access.</strong> Once anyone can ask, someone has to decide what the assistant is allowed to see.</li></ul><p>Better models will pick the right table more often. They will not settle whether a trial user is a customer, because that is a choice, not a fact in the data.</p><p><a href="https://www.linkedin.com/blog/engineering/infrastructure/inside-linkedins-data-context-layer-curating-trustworthy-context-for-humans-and-agents/">LinkedIn’s engineering team describes this work at scale</a>. Teams there write down business definitions, exceptions and known data problems, and owners review them the way they review code.</p><h3>So does the data work grow or shrink?</h3><p>My view is that the shape changes more than the size. The typing shrinks and the judging grows. Whether the total grows depends on three things:</p><ol><li><strong>How many people ask.</strong> One founder asking creates little review work. A sales team of ten asking daily creates a lot.</li><li><strong>How much money depends on the answers.</strong> A wrong number in a team chat costs little. A wrong number in a board deck or a pricing decision costs a lot.</li><li><strong>How many definitions are contested.</strong> If sales, finance and product each count customers differently, every answer carries that dispute.</li></ol><p>Ten people asking daily, or a number that goes to the board, is high. One founder asking weekly is low.</p><p>If all three are low, AI saves you data work, and you need to do nothing more than keep a short glossary. If one is high, name an owner for the judgment. If two or three are high, make that owner a named person with set hours each week. Without an owner, the review falls to whichever engineer knows the data best.</p><h3>What does this look like at each stage?</h3><p>I think the ownership should grow in this order.</p><p><strong>Before launch.</strong> Few people ask and little money depends on the answers. Write a one page glossary: what a customer is, what an active user is, and which accounts are tests. Then ask the AI freely.</p><p><strong>First paying customers.</strong> Money now moves, and sales, support and marketing start asking. This is usually when one of the three conditions turns high. Name an owner for each definition, choose the trusted source for revenue, and start a short file of real questions with answers you have checked. In my experience, a few hours a week of an experienced person, part-time or fractional, is often enough. That person gets the time by no longer taking routine requests the assistant now answers well.</p><p><strong>Growing with a first data person.</strong> The data person owns the definitions, the question file and the review of answers. They also publish which questions the assistant handles reliably and which still need a person.</p><h3>What goes wrong, and how do you catch it?</h3><p><strong>Nobody owns the definition.</strong> Two colleagues ask for active customers and get different numbers, because each phrasing led the assistant to a different meaning.<br>Guardrail: one named owner per definition, kept in a written file that the assistant must cite.<br>Check: have a few colleagues ask the same question in their own words. The numbers should match and name the same definition.</p><p><strong>The review falls to an engineer by accident.</strong> Nobody trusts the replies, so each one goes to the person who knows the data, in addition to their main work.<br>Guardrail: a named reviewer, plus a short public list of question types marked trusted or needs review.<br>Check: ask that person how much of last week went to checking data answers. If they cannot say, nobody is watching the load.</p><p><strong>A confident wrong number reaches a board meeting.</strong> Picture a deck whose revenue growth includes test accounts and refunds. The board plans hiring around growth that is not there.<br>Guardrail: a person traces every number that leaves the company to its definition and its source.<br>Check: before the next board meeting, trace a few numbers from the deck to a written definition and to billing. Remove any number you cannot trace.</p><h3>For your data team</h3><p>This part gets technical. In my view the role moves from answering requests to owning three written things: context, an answer-check file and a review loop. The <a href="https://goagentdata.com/posts/data-agent-harness">harness post</a> covers the same idea from the agent side.</p><p><strong>Context.</strong> One small file per definition, changed by reviewed pull request, that people and agents both read:</p><pre># metrics/active_customer.yaml<br>name: active_customer<br>meaning: account with at least one paid invoice dated in the calendar month (invoices.paid_at)<br>counts: status = paid and amount &gt; 0; excludes is_test and is_internal<br>period: calendar month, UTC<br>source: billing_export.invoices<br>owner: &lt;named person&gt;<br>version: 3<br>changed: v3, refunded invoices no longer count as paid<br>restated: no, past months stay as reported<br>decided_by: &lt;who settled the dispute, and when&gt;</pre><p>The decided_by line matters most. It records that a person made a business choice, so the next disagreement begins with that decision, not with a new debate.</p><p><strong>Answer-check file.</strong> Real past questions with checked answers, each pinned to a date and a definition. It is the test for every definition file. The <a href="https://goagentdata.com/posts/evals-for-data-answers">follow-up post</a> shows how to build and run one.</p><p><strong>Access.</strong> Give the assistant a read-only login limited to the tables you approve. Then a question from sales cannot reach salaries or customer personal data.</p><h3>Check it in 10 minutes</h3><ol><li>Ask your assistant how many active customers you had last month. Ask again in different words. Compare the numbers.</li><li>Ask where “active customer” is defined and who decided it. If nobody can say, you have found your first job.</li><li>Score your company on the three conditions: how many people ask, how much money depends on the answers, how many definitions are contested.</li><li>Pick one number from your last board or investor update and trace it to billing.</li></ol><p>No assistant yet? Do steps 2 to 4.</p><p>See where your setup stands with the <a href="https://goagentdata.com/#self-check">two minute self-check</a>. Then read the <a href="https://goagentdata.com/guide#checks">draft checks</a> your coding agent can run.</p><p><em>Originally published at </em><a href="https://goagentdata.com/posts/ai-analytics-more-data-work"><em>https://goagentdata.com</em></a><em> on October 25, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19fa79d24692" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI is only as good as the data and context you give it]]></title>
            <link>https://medium.com/@baruchjacob/your-ai-is-only-as-good-as-the-data-and-context-you-give-it-5a8344343dcf?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a8344343dcf</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 11:31:25 GMT</pubDate>
            <atom:updated>2026-10-21T11:31:25.483Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR.</strong> “Garbage in, garbage out” used to be the whole rule. With AI it is no longer enough: correct data is necessary but not sufficient. The second condition is context: written definitions, what was recorded and when, whether history was kept, and what the business means by a number. That context used to live in an analyst’s head. An agent has no such head, so the context has to be written down. This week, check three things: is it recorded at all, is the history kept, and is the meaning written down.</p><h3>What changed about “garbage in, garbage out”?</h3><p>For decades the rule was simple: false input, false output. AI adds a second way to fail. Entirely correct data can still give a wrong answer, because the data does not say what it means. A column named “status” holds the word “active” in thousands of rows, and nothing in the column tells the AI whether active means paying, logged in, or merely not deleted. The AI picks one and answers in the same calm tone either way.</p><p>Until recently, a business analyst carried that context. They knew which statuses to leave out, which month the tracking broke, and which customer was a pilot. When they forgot, they searched old chat threads or asked someone what happened when. The context was real, but it lived in a person. That was always a weak spot: when the analyst left, the context left too. Agents make it impossible to ignore.</p><p>An agent cannot ask around. When something is missing, it fills the gap with a reasonable assumption, and it rarely says it made one. A reasonable assumption about your business is still a guess. So the context has to be written where the agent can read it.</p><p>I think of it as two necessary conditions: correct data first, context second. Neither replaces the other.</p><h3>Is a better model the fix?</h3><p>It helps, but only up to a point. A stronger model explores more: it may list every value in a status column and notice one it did not expect. In short, an agent can discover your data, not your intent. No model can know a rule that was never written down, such as which customers your team treats as pilots. And the strongest models are available to your competitors too. Your data and its context are the parts that are yours to improve.</p><h3>What does missing context look like?</h3><p>Imagine a warehouse where “items in stock” is defined as every item that has not exited. It sounds right, and every row behind it is correct. But some items are marked missing, and some are scrapped. Neither group ever exited, so both sit inside the stock count, although they are on no shelf.</p><p>The warehouse team knows this. Ask anyone on the floor and they will tell you missing items are gone and scrapped items are written off. The analyst who built the report knew it too, and quietly filtered them out in the spreadsheet. That knowledge just never reached the written definition.</p><p>So the data is correct and the knowledge exists. What is missing is the link between them. An agent asked “how many items are in stock?” cannot ask the floor. It counts every item that has not exited, and answers with a confident number that is too high.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SweDT_LbBxkAO5zATkrJbQ.jpeg" /></figure><p><em>Before, the assistant counts every item that never exited, missing and scrapped ones included, and answers at once with a confident number that is too high. After, a context card on the path defines in stock and what it excludes, so the same question returns a lower number with the definition it used.</em></p><h3>What does context actually include?</h3><p>Context is four plain things:</p><ul><li><strong>Definitions.</strong> What the business means by “in stock”, “customer”, “active” or “revenue”, and what is excluded.</li><li><strong>What was recorded, and when.</strong> A field added in March says nothing about January. A change in how you record something creates a break in the trend.</li><li><strong>Whether history was kept.</strong> Did the system keep each past state, or does it hold only today’s?</li><li><strong>What a number is for.</strong> The decision it informs and the person who owns it.</li></ul><h3>What can a non-technical founder check this week?</h3><p>Each check pairs a failure with its guardrail, and none needs code.</p><p><strong>1. Is it recorded at all?</strong><br>Failure: the AI answers a question about something your systems never captured, so it infers or guesses.<br>Guardrail: for each question you care about, name the system that records it, and the person who owns it.<br>Check: pick three questions you ask often, such as stock levels, lost customers or sales by channel. For each, ask your team which system holds the answer. If nobody can name one, the AI is guessing. And if nobody knows where the data lives, nobody can take responsibility for it either.</p><p><strong>2. Is history kept?</strong><br>Failure: a record is overwritten when something changes, so the past is gone and any trend is quietly wrong.<br>Guardrail: keep each change as a new row with a date, or keep a dated copy, instead of editing in place.<br>Check: ask your developer to show you a record that changed last month. Can you see what it was before? If you only see today’s value, history is not kept.</p><p><strong>3. Is the meaning written down?</strong><br>Failure: a number has a definition only in someone’s head, so each reader, human or AI, invents its own.<br>Guardrail: one short written definition for each key number, including what it excludes, stored where your AI can read it.<br>Check: ask your AI assistant “which definition did you use for that number, and where is it written?” Then open the document it names. If there is no document, or it does not say what the AI claimed, the meaning was its own guess.</p><h3>What should you do at each stage?</h3><p><strong>Before launch.</strong> Decide what you will need to answer later and make sure the product records it from day one. Write a one page glossary with the one or two numbers that matter, and keep history from the start.</p><p><strong>First paying customers.</strong> Run the three checks above against the production database, billing and your CRM (where sales tracks customers). Give each key number its own short written definition in a shared folder, and point any AI you use at that folder. Name one person who owns it, even if it takes them an hour a week.</p><p><strong>Growing with a first data person.</strong> Add review. A change to a definition or to what is recorded gets a named approver and a dated note. The first data person owns the definitions and that change log, because the judgment about which context matters is the hard part.</p><h3>For your data team</h3><p>This part gets technical. The pattern is simple: treat context as data. Keep it versioned, queryable and next to the thing it describes.</p><p><strong>Pick the right slowly changing dimension type, per column.</strong> A slowly changing dimension is an attribute that changes now and then, such as an item’s status, a customer’s plan or a product’s price. How you store the change decides what history you can answer later.</p><ul><li><strong>Type 1, overwrite.</strong> Keep only the current value. Right for corrections, like a typo in a name, where the old value was never true.</li><li><strong>Type 2, new row per change.</strong> Each version gets valid_from, valid_to and is_current (the <a href="https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/type-2/">Kimball Group&#39;s description</a> is the classic reference). Right for anything you will trend, reconcile or audit: status, plan, location, price. This is the default for business state.</li><li><strong>Type 3, previous value column.</strong> Keeps one step back. Rarely enough, because the second change erases the first.</li><li><strong>Periodic snapshot.</strong> A dated copy of the whole table each day or month. Simple, and good for stock levels and balances, but it misses changes between snapshots.</li></ul><p>The choice is per column, not per table. In the warehouse case, the item’s name can be Type 1, while its status must be Type 2, or “in stock last quarter” cannot be answered at all.</p><p><strong>Give the agent a rule, and enforce it separately.</strong> Instruct it to use only documented definitions, to cite the file behind every number and to say when none exists. Enforce the boundary with read-only credentials on a copy of the data, never the live database. A copy that mirrors only the current state still holds no history, so choose the type in the source.</p><pre>metric: items_in_stock<br>definition: items physically available in the warehouse<br>include: status = received and not exited<br>exclude:<br>  - status = missing      # never exited, not on a shelf<br>  - status = scrapped     # written off, never exited<br>history: item_status is type 2 (valid_from, valid_to, is_current)<br>recorded_since: 2026-03-01    # example: no status history before this date<br>owner: head of operations</pre><h3>Check it in 10 minutes</h3><ol><li>Pick one question you would ask an AI about your business, such as “how many items are in stock?”</li><li>Ask your AI and write down the answer.</li><li>Ask which definition it used, what it excludes, and where it is written. If the answer is “nowhere”, write it today.</li><li>Ask whether the system kept last quarter’s history. If it only holds today’s state, note that gap.</li><li>Ask the person who knows the area best for the same answer. If the two differ, the difference is your missing context.</li></ol><p>See where your setup stands with the <a href="https://goagentdata.com/#self-check">two minute self-check</a>, or read the <a href="https://goagentdata.com/guide#checks">draft checks</a> your coding agent can run.</p><p><em>Originally published at </em><a href="https://goagentdata.com/posts/ai-needs-data-and-context"><em>goagentdata.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a8344343dcf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Healing Pipelines with DBT: The Observe & Fix Method ]]></title>
            <link>https://medium.com/@baruchjacob/self-healing-pipelines-with-dbt-the-observe-fix-method-9d6b2da4eae3?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d6b2da4eae3</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 19:35:37 GMT</pubDate>
            <atom:updated>2025-04-14T05:57:27.219Z</atom:updated>
            <content:encoded><![CDATA[<h3>Observe &amp; Fix — A Practical Data Quality Pattern for DBT + Snowflake🔎🛠</h3><p>As AI and automated systems increasingly consume and act on organizational data, clean pipelines are no longer just a “nice to have” — they’re mission critical. Your models aren’t just feeding dashboards anymore — they’re driving decisions, predictions, and customer experiences in real time.</p><p>That’s why data quality matters more than ever.</p><p>Data quality isn’t a checkbox. It’s a continuous discipline. Ignore it early — and you’ll pay for it later.</p><p>In this post, I’m sharing a method I’ve used to manage “fixable” data quality issues inside <a href="https://www.getdbt.com/product/what-is-dbt">DBT</a>: <strong>Observe &amp; Fix</strong> — a simple form of self-healing for your pipelines, without the complexity or buzzwords.</p><p>It’s a clear, two-step pattern: first, <strong>observe</strong> potential data problems like duplicates, missing values, or bad logic. Then, <strong>fix</strong> them in downstream models — without breaking your pipeline or hiding issues.</p><p>This method doesn’t require any new tools — just DBT models, tests, and macros.</p><p>Of course, pairing it with observability tools like <a href="https://www.elementary-data.com/home">Elementary</a> can enhance alerting and visibility — but it’s not required.</p><h3>🧠 What is Observe &amp; Fix?</h3><ul><li><strong>Observe</strong> — detect data quality issues early by tagging them visibly in staging models. This can be done via simple tests or reusable macros.</li><li><strong>Fix</strong> — apply logic to deduplicate, filter, or correct issues in your core models.</li></ul><p>It’s a lightweight approach that helps you move fast <strong>without hiding dirty data</strong>.<br>(<em>Shift Left principle makes a guest appearance here — fix issues closer to the source, not downstream. </em><a href="https://medium.com/p/2ca5460c1877"><em>More on that here</em></a>)</p><h3>🤔 Haven’t We Seen This Before?</h3><p>Sure — similar ideas appear in Great Expectations, Elementary, Monte Carlo, and DBT threads. But what’s often missing is a simple, repeatable way to implement it natively in DBT.</p><p>So here it is: clear, modular, and ready for your project.</p><h3>⚙️ Example: Detecting and Fixing Duplicate Orders</h3><p>Let’s say we have a raw source orders, and a staging model stg_orders:</p><pre>-- models/bronze/stg_orders.sql<br>select<br>  id,<br>  user_id,<br>  created_at,<br>  amount<br>from {{ source(&#39;raw&#39;, &#39;orders&#39;) }}</pre><p>You want to make sure id is unique. There are two ways to handle this:</p><h3>🔍 Observation: Detecting Issues Early</h3><p>Here are three practical ways to implement the “Observe” step. Choose based on the level of visibility, effort, and reuse you need.</p><h3>✅ Option 1: Use a simple test with severity: warn</h3><p>Use when:</p><ul><li>You want minimal observability</li><li>You don’t need the actual duplicate rows</li><li>You want DBT to surface the issue passively</li></ul><p>In schema.yml:</p><pre>models:<br>  - name: stg_orders<br>    description: &quot;Staging layer for orders&quot;<br>    tags: [staging, observe]<br>    columns:<br>      - name: id<br>        tests:<br>          - unique:<br>              severity: warn<br>          - not_null</pre><h3>✅ Option 2: Build a dedicated observability model with a macro</h3><p>💬 <em>Want to turn this into an alert? If you’re using Elementary (open source), it can automatically detect when this model returns rows and trigger a Slack or email alert. Just add </em><em>alert: true in your Elementary config for this model.</em></p><p>Use when:</p><ul><li>You want to <strong>see the actual bad rows</strong></li><li>You want to query or log issues</li><li>You want the observability logic to be reusable</li></ul><pre>-- macros/observe/observe_duplicates_advanced.sql<br>{% macro observe_duplicates_advanced(model_ref, columns, check_name=&#39;duplicate_check&#39;, error_code=&#39;DUPLICATE_ROW&#39;) %}<br>    {% set col_list = columns | join(&#39;, &#39;) %}<br>    {% set partition_by = columns | join(&#39;, &#39;) %}<br><br>    with base as (<br>        select *,<br>            count(*) over (partition by {{ partition_by }}) as dup_count,<br>            row_number() over (partition by {{ partition_by }} order by {{ partition_by }}) as rn<br>        from {{ ref(model_ref) }}<br>    )<br><br>    select <br>        {{ col_list }},<br>        dup_count,<br>        &#39;{{ check_name }}&#39; as check_name,<br>        &#39;{{ error_code }}&#39; as error_code,<br>        current_timestamp() as check_timestamp<br>    from base<br>    qualify dup_count &gt; 1 and rn = 1<br>{% endmacro %}</pre><p>Used in a model:</p><pre>-- models/observe/orders_observe_duplicates.sql<br>{{ observe_duplicates_advanced(&#39;stg_orders&#39;, [&#39;id&#39;], &#39;orders_id_dup_check&#39;, &#39;ORDERS_DUPLICATE_ID&#39;) }}</pre><p>In schema.yml:</p><pre>models:<br>  - name: orders_observe_duplicates<br>    description: &quot;Detects duplicate order IDs using a macro&quot;<br>    tags: [observe]</pre><h3>✅ Option 3: Turn the macro into a custom test</h3><p>🗂️ <em>How this works: This creates a “custom test” in DBT that runs during </em><em>dbt test. If the query returns rows, DBT marks the test as a warning or error depending on how you configure it.</em></p><p>Use when:</p><ul><li>You want test results to show up in DBT Cloud or CLI runs</li><li>You want warnings for duplicates, but don’t need a model</li><li>You prefer to integrate with DBT-native test handling</li></ul><p>Here’s how to define the test:</p><pre>-- tests/expect_no_duplicates.sql<br>{% test expect_no_duplicates(model, column_name) %}<br>select {{ column_name }}<br>from {{ model }}<br>group by {{ column_name }}<br>having count(*) &gt; 1<br>{% endtest %}</pre><p>In schema.yml:</p><pre>models:<br>  - name: stg_orders<br>    columns:<br>      - name: id<br>        tests:<br>          - expect_no_duplicates:<br>              column_name: id<br>              severity: warn</pre><h3>🛠 Step 2: Fix the Problem</h3><p>Once the issues are observed, we fix them in the next model.</p><p>In this case, we deduplicate using QUALIFY:</p><pre>-- models/silver/orders.sql<br>select <br>  id,<br>  user_id,<br>  created_at,<br>  amount<br>from {{ ref(&#39;stg_orders&#39;) }}<br>qualify row_number() over (partition by id order by updated_at desc) = 1</pre><p>This approach lets you decide how to resolve conflicts — whether by keeping the latest row, coalescing fields, or filtering out bad data entirely.</p><p>You can also tag this model:</p><pre>models:<br>  - name: orders<br>    description: &quot;Core orders model with deduplication&quot;<br>    tags: [fix, core]</pre><h3>🤖 But Isn’t Self-Healing Reserved for AI?</h3><p>We often hear “self-healing pipelines” and imagine complex ML-driven anomaly detection or auto-remediation systems.</p><p>But here’s the truth: in most real-world cases, resilience doesn’t come from algorithms — it comes from <strong>thoughtful model design</strong>.</p><p>When you:</p><ul><li>define observability logic up front (like severity: warn or a macro to catch duplicates), and</li><li>apply clean, rule-based transformations in downstream models.</li></ul><p>you get a self-healing pipeline (for “fixable” issues) — without needing external orchestration or reactive alerts.</p><p>✅ Great for known, repeatable issues (e.g., duplicates, nulls)<br>⚠️ Not a replacement for monitoring tools — but a complementary method</p><p><strong>You define the rules once. The models keep themselves clean.</strong></p><p>That’s what we mean by self-healing — without the buzzwords.</p><h3>🧭 Why This Pattern Works</h3><ul><li>✅ Keeps pipelines flowing (no test failures — where not needing)</li><li>✅ Still surfaces quality issues clearly</li><li>✅ Encourages proactive treatment of dirty data</li><li>✅ Works natively in DBT, with no extra tools</li></ul><p>It’s about separating <strong>detection from remediation</strong>, and being honest about the data quality challenges that show up in real-world systems.</p><h3>🧪 Try It Yourself</h3><p>You don’t need a new tool or platform. Just a good habit:</p><ul><li>Add lightweight observe logic in your staging layer</li><li>Use macros or custom tests for reusability</li><li>Handle issues intentionally in your fix models</li></ul><p>That’s Observe &amp; Fix. Simple. Transparent. And scalable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBvUTs0oTycuAoa6rqdJag.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d6b2da4eae3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Robust Snowflake Calendar UDF: Simplifying Date Management for Analytics & Reporting ]]></title>
            <link>https://medium.com/@baruchjacob/building-a-robust-snowflake-calendar-udf-simplifying-date-management-for-analytics-reporting-86d656362797?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/86d656362797</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 08:46:01 GMT</pubDate>
            <atom:updated>2025-02-12T08:46:01.631Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s data-driven world, managing dates efficiently is essential for building scalable and reliable data pipelines. <br>Aligning fiscal periods, handling business days, and incorporating holidays are common challenges that often lead to complex and hard-to-maintain SQL logic.</p><p>The <a href="https://github.com/jacobbaruch/calendar-udf-snowflake"><strong><em>Snowflake Calendar UDF</em></strong></a> simplifies date management by generating a calendar table for any date range. It includes support for fiscal year logic, holiday metadata, and multiple time granularities, providing a single reliable source for all your date-related data.</p><h3>Why Use a Calendar UDF?</h3><p>When building data pipelines, it’s critical to have a consistent and reliable source for date logic. Common issues include:</p><ul><li>Complex fiscal calendars that don’t align with standard months.</li><li>Handling business days versus weekends and holidays.</li><li>Inconsistent time granularities (minute, hour, day, week, month, etc.).</li><li>Redundant date logic across multiple models and tables.</li></ul><p>This UDF ensures consistency by providing a single place for all date-related data, eliminating the need for each developer and analyst to generate their own calendar logic.</p><h3>How the UDF Works</h3><p>The UDF generates a calendar table based on three input parameters:</p><pre>Column Name   | Data Type  | Description<br>--------------|------------|-------------------------------<br>from_ts       | TIMESTAMP  | The starting timestamp for generating the calendar.<br>to_ts         | TIMESTAMP  | The ending timestamp for generating the calendar.<br>time_type     | STRING     | The granularity of the calendar. <br>              |            |Supported values: &#39;minute&#39;, &#39;hour&#39;, &#39;day&#39;, &#39;week&#39;, &#39;month&#39;, &#39;quarter&#39;, &#39;year&#39;.</pre><h3>Example Query:</h3><pre>SELECT *<br>FROM TABLE(<br>    YOUR_DATABASE.YOUR_SCHEMA.generate_calendar(<br>        TIMESTAMP &#39;2023-01-01 00:00:00&#39;, <br>        TIMESTAMP &#39;2023-12-31 23:59:59&#39;, <br>        &#39;day&#39;<br>    )<br>);</pre><h3>What You Get: Output Columns</h3><p>The generated calendar table includes the following useful fields:</p><pre>Column Name       | Data Type   | Description<br>------------------|-------------|-----------------------------------------------<br>timestamp         | TIMESTAMP   | The exact timestamp for the date.<br>date              | DATE        | The full date (YYYY-MM-DD).<br>year              | INT         | The calendar year.<br>quarter           | INT         | The calendar quarter (1–4).<br>month             | INT         | The calendar month (1–12).<br>day               | INT         | The day of the month (1–31).<br>week              | INT         | The ISO week number of the year (1–52).<br>day_of_week       | INT         | The day of the week (1 = Sunday, 7 = Saturday).<br>day_name          | STRING      | The name of the day (e.g., Monday, Tuesday).<br>is_weekend        | BOOLEAN     | Indicates if the date is a weekend (TRUE/FALSE).<br>is_business_day   | BOOLEAN     | Indicates if the date is a business day (TRUE/FALSE).<br>fiscal_year       | INT         | The fiscal year associated with the date.<br>fiscal_quarter    | INT         | The fiscal quarter (1–4).<br>is_holiday        | BOOLEAN     | Indicates if the date is a holiday (TRUE/FALSE).<br>holiday_name      | STRING      | The name of the holiday (if applicable).<br>holiday_day_type  | STRING      | The type of holiday (e.g., Public, Religious).</pre><h3>Why This Matters for Data Pipelines</h3><p>Centralizing date logic in a single UDF simplifies your data models and ensures consistent calculations across all pipelines. Use this UDF for:</p><ul><li><strong>Accurate Business Metrics</strong> — Filter data by business days.</li><li><strong>Financial Reporting</strong> — Align data with fiscal calendars.</li><li><strong>Holiday Impact Analysis</strong> — Adjust for holiday-specific performance.</li></ul><h3>Integrating with DBT</h3><p>For those using DBT, integrating this UDF is simple. Use the provided dbt_set_source.yml file to configure it as a source. This ensures all your data models reference the same reliable calendar.<br>Example DBT Model:</p><pre>SELECT *<br>FROM TABLE(<br>    {{ source(&#39;calendar_udf&#39;, &#39;generate_calendar&#39;) }}(<br>        TIMESTAMP &#39;2023-01-01 00:00:00&#39;, <br>        TIMESTAMP &#39;2023-12-31 23:59:59&#39;, <br>        &#39;day&#39;<br>    )<br>);</pre><h3>Better Pipelines Start with Better Tools</h3><p>A reliable calendar table is essential for building scalable data pipelines. The <strong>Snowflake Calendar UDF</strong> helps automate and streamline date-related logic, saving time and reducing complexity.</p><p><a href="https://github.com/jacobbaruch/calendar-udf-snowflake"><strong><em>👉 Check out the repository and get started today</em></strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*1F3aO99g7EF_RIL1Oi6yAw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86d656362797" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Critical Principles for Modern Data Modeling]]></title>
            <link>https://medium.com/@baruchjacob/three-critical-principles-for-modern-data-modeling-2ca5460c1877?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ca5460c1877</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-transformation]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Mon, 06 Jan 2025 00:27:43 GMT</pubDate>
            <atom:updated>2025-01-09T21:02:07.804Z</atom:updated>
            <content:encoded><![CDATA[<p>As we advance into an era where AI agents and automated tools increasingly handle data analysis, the fundamentals of proper data modeling become more critical than ever. While these tools can analyze data more efficiently, their effectiveness heavily depends on how well we structure and prepare our data.</p><p>The principles outlined here aren’t just guidelines or best practices — they’re essential requirements for building reliable, scalable, and efficient data systems. Without them, even the most sophisticated AI tools will struggle to deliver consistent and actionable insights.</p><h4>Why These Principles Matter More Than Ever</h4><p>The modern data landscape is evolving rapidly:</p><ul><li><strong>Exponential growth in data volume and complexity</strong>: Businesses are ingesting and managing more data than ever before.</li><li><strong>Increasing reliance on automated analysis tools</strong>: Machine learning and AI are reshaping how decisions are made.</li><li><strong>Rising importance of data quality and consistency</strong>: Stakeholders demand reliable insights, not excuses for inaccuracies.</li><li><strong>The need for scalability</strong>: Systems must handle today’s needs while adapting to tomorrow’s challenges.</li></ul><p>These trends make proper data modeling critical. Without a robust framework, data pipelines become fragile, insights unreliable, and governance unsustainable. The principles we’ll discuss offer a roadmap to tackle these challenges head-on.</p><p><em>Note: All examples in this post use DBT &amp; Snowflake syntax for clarity and consistency, but the principles and patterns apply to any modern data stack and tools.</em></p><h4>Principle 1: Shift Left — Early Definitions</h4><p><strong>What is it?</strong></p><p>The “Shift Left” principle emphasizes defining key transformations and metrics as early as possible in your pipeline. This approach:</p><ul><li>Prevents inconsistencies across downstream models.</li><li>Enables thorough QA at every layer.</li><li>Simplifies data lineage, making pipelines easier to debug.</li><li>Promotes reusability across teams and domains.</li></ul><p>Multiple teams can rely on the same core logic by defining metrics and dimensions early, reducing redundancy and encouraging collaboration.</p><p><strong>Example 1: User Behavior Analysis<br></strong>Reusable tables defined early in the pipeline simplify analysis across domains:</p><pre>-- models/bronze/user_events.sql<br>-- Clean individual event data at the bronze level<br>SELECT<br>  user_id,<br>  event_type,<br>  session_id,<br>  event_timestamp::date AS event_date<br>FROM {{ source(&#39;raw&#39;,&#39;events&#39;) }}<br><br>-- models/silver/user_engagement_daily.sql<br>-- Reusable user engagement metrics<br>SELECT<br>  user_id,<br>  event_date,<br>  count(distinct iff(event_type = &#39;page_view&#39;, session_id, null)) as daily_sessions,<br>  count(distinct iff(event_type = &#39;product_view&#39;, product_id, null)) as products_viewed<br>FROM {{ ref(&#39;bronze_user_events&#39;) }}<br>GROUP BY 1,2;</pre><p><strong>Example 2: Marketing Attribution<br></strong>Early definitions make cross-domain attribution straightforward:</p><pre>-- models/silver/customers_with_details.sql<br>-- Define customer details<br>SELECT<br>  c.customer_id,<br>  c.customer_name,<br>  d.loyalty_status<br>FROM {{ ref(&#39;bronze_customers&#39;) }} c<br>JOIN {{ ref(&#39;bronze_customer_details&#39;) }} d<br>  ON c.customer_id = d.customer_id;<br><br>-- models/gold/customer_lifetime_value.sql<br>-- Use defined customer details for CLV analysis<br>SELECT<br>  cd.customer_id,<br>  cd.customer_name,<br>  cd.loyalty_status,<br>  SUM(t.transaction_amount) AS total_spend,<br>  COUNT(t.transaction_id) AS total_transactions<br>FROM {{ ref(&#39;silver_customers_with_details&#39;) }} cd<br>JOIN {{ ref(&#39;bronze_transactions&#39;) }} t<br>  ON cd.customer_id = t.customer_id<br>GROUP BY 1,2,3;</pre><p><a href="https://euno.ai/">Euno </a>is a very interesting product I found for achieving this principle retrospectively.</p><h4>Principle 2: The Single Join Rule</h4><p><strong>What is it?</strong></p><p>The Single Join Rule ensures each table is joined only once in the entire transformation process<em>for the same intent or logic</em>. This helps avoids:</p><ul><li>Data inconsistencies from multiple join paths.</li><li>Query performance issues due to redundant joins.</li><li>Complex and unmaintainable pipelines.</li></ul><p>By defining relationships once, you maintain clarity and ensure that all downstream models rely on the same source of truth.</p><p><strong>Example 1: Product Performance Metrics<br></strong>Ensure relationships between products and regions are defined once and reused consistently:</p><pre>-- models/silver/product_regions.sql<br>-- Define product-region relationships<br>SELECT<br>  p.product_id,<br>  p.product_name,<br>  r.region_id,<br>  r.region_name<br>FROM {{ ref(&#39;bronze_products&#39;) }} p<br>JOIN {{ ref(&#39;bronze_regions&#39;) }} r<br>  ON p.region_id = r.region_id;<br><br>-- models/gold/product_sales_analysis.sql<br>-- Use the product-region relationship<br>SELECT<br>  pr.product_id,<br>  pr.product_name,<br>  pr.region_name,<br>  SUM(sales_amount) AS total_sales,<br>  COUNT(order_id) AS total_orders<br>FROM {{ ref(&#39;silver_product_regions&#39;) }} pr<br>JOIN {{ ref(&#39;bronze_sales&#39;) }} s<br>  ON pr.product_id = s.product_id<br>GROUP BY 1,2,3;</pre><p><strong>Example 2: Customer Lifetime Value (CLV)<br></strong>Combine customer details with transaction history to calculate CLV:</p><pre>-- models/silver/customers_with_details.sql<br>-- Define customer details<br>SELECT<br>  c.customer_id,<br>  c.customer_name,<br>  d.loyalty_status<br>FROM {{ ref(&#39;bronze_customers&#39;) }} c<br>JOIN {{ ref(&#39;bronze_customer_details&#39;) }} d<br>  ON c.customer_id = d.customer_id;<br><br>-- models/gold/customer_lifetime_value.sql<br>-- Use defined customer details for CLV analysis<br>SELECT<br>  cd.customer_id,<br>  cd.customer_name,<br>  cd.loyalty_status,<br>  SUM(t.transaction_amount) AS total_spend,<br>  COUNT(t.transaction_id) AS total_transactions<br>FROM {{ ref(&#39;silver_customers_with_details&#39;) }} cd<br>JOIN {{ ref(&#39;bronze_transactions&#39;) }} t<br>  ON cd.customer_id = t.customer_id<br>GROUP BY cd.customer_id, cd.customer_name, cd.loyalty_status;</pre><h4><strong>Principle 3: Data Domain Encapsulation</strong></h4><p><strong>What is it?</strong><br>Data domain encapsulation ensures each domain has a single source of truth. <br>Encapsulation:</p><ul><li>Promotes consistency and reusability: Each domain encapsulates its transformations, ensuring that all downstream models use a unified logic.</li><li>Simplifies governance and lineage tracking: Ownership and accountability are clear, making it easier to track changes and ensure compliance.</li><li>Reduces the risk of cascading errors: Updates and changes are contained within the domain, minimizing their impact on other areas.</li></ul><p>By isolating domain-specific transformations, teams can independently manage their logic while ensuring alignment across the organization.</p><p><strong>Example 1: Finance Data Domain - Revenue Reporting</strong></p><pre>-- models/bronze/finance_transactions.sql<br>-- Standardize transaction data<br>SELECT<br>  transaction_id,<br>  customer_id,<br>  transaction_dttm,<br>  transaction_dttm::date as transaction_date,<br>  revenue_amount,<br>  tax_amount<br>FROM {{ source(&#39;raw&#39;,&#39;transactions&#39;) }}<br>WHERE revenue_amount &gt; 0;<br><br>-- models/bronze/customers.sql<br>-- Standardize customer data<br>SELECT<br>  customer_id,<br>  customer_name,<br>  signup_dttm,<br>  signup_dttm::date as signup_date,<br>  loyalty_status<br>FROM {{ source(&#39;raw&#39;,&#39;customers&#39;) }}<br><br><br>-- models/silver/finance_revenue.sql<br>-- Combine transaction and customer data<br>SELECT<br>  t.transaction_id,<br>  t.transaction_date,<br>  t.revenue_amount,<br>  t.revenue_amount - t.tax_amount AS net_revenue,<br>  c.customer_id,<br>  c.customer_name,<br>  c.loyalty_status<br>FROM {{ ref(&#39;bronze_finance_transactions&#39;) }} t<br>JOIN {{ ref(&#39;bronze_customers&#39;) }} c<br>  ON t.customer_id = c.customer_id;<br><br>-- models/gold/finance_kpi_summary.sql<br>-- Summarize key performance indicators<br>SELECT<br>  transaction_date,<br>  SUM(revenue_amount) AS total_revenue,<br>  SUM(net_revenue) AS total_net_revenue,<br>  COUNT(DISTINCT customer_id) AS unique_customers<br>FROM {{ ref(&#39;silver_finance_revenue&#39;) }}<br>GROUP BY transaction_date;</pre><p><strong>Example 2: Marketing Data Domain<br></strong>Encapsulate touchpoints and conversions:</p><pre>-- models/bronze/marketing_touchpoints.sql<br>-- Standardize touchpoint data<br>SELECT<br>  user_id,<br>  campaign_id,<br>  touch_dttm,<br>  touch_dttm::date AS touch_date,<br>  channel<br>FROM {{ source(&#39;raw&#39;, &#39;marketing_touchpoints&#39;) }};<br><br>-- models/bronze/marketing_conversions.sql<br>-- Standardize conversion data<br>SELECT<br>  user_id,<br>  campaign_id,<br>  conversion_dttm,<br>  conversion_dttm::date AS conversion_date,<br>  conversion_value<br>FROM {{ source(&#39;raw&#39;, &#39;marketing_conversions&#39;) }}<br>WHERE conversion_value &gt; 0;<br><br>-- models/silver/marketing_performance.sql<br>-- Combine touchpoints and conversions<br>SELECT<br>  t.campaign_id,<br>  t.touch_date,<br>  COUNT(DISTINCT t.user_id) AS users_reached,<br>  COUNT(DISTINCT c.user_id) AS conversions,<br>  SUM(c.conversion_value) AS total_revenue<br>FROM {{ ref(&#39;bronze_marketing_touchpoints&#39;) }} t<br>LEFT JOIN {{ ref(&#39;bronze_marketing_conversions&#39;) }} c<br>  ON t.user_id = c.user_id<br>  AND t.campaign_id = c.campaign_id<br>  AND t.touch_date &lt;= c.conversion_date<br>GROUP BY 1,2;</pre><p>By adopting <strong>Shift Left</strong>, <strong>The Single Join Rule</strong>, and <strong>Data Domain Encapsulation</strong>, you will:</p><ul><li><strong>Improve QA</strong>: Easier validation and debugging with reliable structures.</li><li><strong>Reduce support needs</strong>: Consistent models empower stakeholders to self-serve.</li><li><strong>Enhance data governance</strong>: Encapsulation ensures clear ownership and traceability.</li><li><strong>Future-proof scalability</strong>: Modular designs adapt seamlessly to evolving needs.</li></ul><p>These principles are foundational for building reliable, scalable, and maintainable data systems that support modern AI-driven decision-making. Proper data modeling isn’t just technical — it’s a critical enabler of business success in an increasingly automated world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEfi6TO6Jk_IqP9hzkH8Ag.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ca5460c1877" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Buzzword to Reality: How LLM Enabling Self-service BI]]></title>
            <link>https://medium.com/@baruchjacob/from-buzzword-to-reality-how-openai-enabling-self-service-bi-a7f2923d110c?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7f2923d110c</guid>
            <category><![CDATA[self-service-bi]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[dbt-labs]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Sun, 12 Mar 2023 20:24:22 GMT</pubDate>
            <atom:updated>2023-07-04T06:02:16.809Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction / What is Self Service BI?</h3><p>There are few definitions of what it is self service BI, you can find below two of my favorites:</p><blockquote><a href="https://www.gartner.com/en/information-technology/glossary/self-service-business-intelligence">Gartner Definition: </a><br>“Self-service business intelligence is defined here as end users designing and deploying their own reports and analyses within an approved and supported architecture and tools portfolio.”</blockquote><blockquote><a href="https://www.techtarget.com/searchbusinessanalytics/definition/self-service-business-intelligence-BI#:~:text=Self%2Dservice%20business%20intelligence%20(BI,data%20mining%20and%20statistical%20analysis.">TechTarget Definition: <br></a>“Self-service business intelligence (BI) is an approach to <a href="https://www.techtarget.com/searchdatamanagement/definition/data-analytics">data analytics</a> that enables business users to access and explore data sets even if they don’t have a background in <a href="https://www.techtarget.com/searchbusinessanalytics/definition/business-intelligence-BI">BI</a> or related functions like data mining and statistical analysis”</blockquote><p>As I see it Self-service BI aims to provide non-technical users with two primary benefits:</p><h4>1. Quick and precise answers without assistance.<br>2. The ability to access and explore data on their own.</h4><p>As the volume and complexity of data continues to grow, and businesses increasingly rely on data-driven decisions, self-service BI has become a crucial aspect of modern data analytics.</p><h3>Why Self Service BI is so important</h3><h4>1. Quick and precise answers without assistance:</h4><p>There a lot of simple questions that are been asked by all departments, roles &amp; levels in any hour of the day. (i.e. <em>“In which product we had the highest sales drop in this month?”, “How many active users we had yesterday?”, “How many open opportunities are stuck?”, ”How many customer complaints did we receive this week?”</em>) getting an answer to any of these questions in no time is crucial as delays in decision-making or incorrect decisions based on incomplete information could lead to negative outcomes.</p><p>getting an answer to this type of questions isn’t required from the person asking the question any skill. and usually in order to get answer for this types of questions, the person need to enter to a specific report / dashboard whether he got it in email, slack or whether he needs the visit an online report / dashboard — any of this is a distraction that not bound to reality.<br>and at the worst case, the person would need an assistant from data person in order to get an answer, which for sure won’t be at real time, and will certainly not be an optimal use of time for both.</p><h4>2. Ability to access and explore data on their own:</h4><p>Self-service BI gives non-technical users the ability to explore data on their own (without waiting for a data analyst to generate reports or analysis) , which can lead to insights and discoveries that they might not have found otherwise. <br>However, While self-service BI can empower non-technical users to explore and analyze data on their own, it also comes with the risk of incorrect or misleading insights, as these users may not have the necessary expertise to properly interpret and analyze the data.</p><h3>Why self-service BI is primarily a buzzword</h3><p>Over the last decade, there have been many buzzwords in the data world, and self-service BI was one of the most frequently heard. However, in many cases, it was just that — a buzzword without much substance behind it. While the idea of<strong> quick and precise answers without assistance</strong> was appealing, the reality was often much messier. For example, imagine being in a meeting and needing to know the expected MRR by the end of the year. If the relevant finance person is there and has a great memory, no problem. But if not, finding the answer could involve navigating to a public report , which may or may not exist and may or may not provide the necessary information. In other cases, getting the answer might involve calling someone and hoping they can provide the information, and if they are not accessible, most of the time we end up continuing without knowing the answer to our desired question.</p><p>While the idea of giving non technical people the <strong>ability to access and explore data on their own</strong> had more coverage and sounded like a great idea, it also missed a lot of crucial aspects. For example, the data might be difficult to understand without proper training or experience in data analysis, leading to incorrect interpretations and decisions. <br>For example, a marketing manager may use self-service BI to analyze website traffic data and draw conclusions about which channels are driving the most traffic. However, without a proper understanding of data analysis, statistical significance and potential biases the manager may misinterpret the data and make incorrect decisions about marketing spend and strategy.</p><p>Furthermore, BI tools can be complex and difficult to use, requiring extensive training and support, which can be a significant investment in time and resources. <br>Overall, while self-service BI has the potential to be a valuable tool, it is important to consider the practical realities and challenges of implementing it effectively.</p><h3>How Self Service BI becoming a reality using NLP</h3><p>From day one when I was exposed to ChatGPT I knew that solutions enabling organizations to get <strong>quick and precise answers without assistance </strong>without the need for data experts to give the answer would soon become a reality.</p><p>While there are already several solutions available that are using LLM abilities, some of which are designed for specific systems like Salesforce, others offer users the ability to ask questions and get answers within their own BI environment. However, it was the <a href="https://www.loom.com/share/4a320cad5c7348cdb686a8e7bbb1144b">video</a> of <a href="https://www.delphihq.com/#about">Delphi</a> that caught my attention due to its unique benefits:<br>1. Offering users the ability to ask questions within a familiar system like slack.<br>2. providing access to complex data model within the organization’s data warehouse.</p><p>Additionally, the facts that the results can be seen and modify at <a href="https://www.lightdash.com/">Lightdash </a>and that Delphi works closely with DBT, makes it even more appealing (even though it’s connected to the Controversial <a href="https://docs.getdbt.com/docs/build/metrics">metrics </a>&amp; semantic layer of DBT)</p><p>Find below two images that illustrate how Delphi is helping us: <br><em>(You’ll need to trust me that the answers were right :))</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*GJjiavXIiKTiMnW66eQl5Q.png" /><figcaption>Example №1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*alkUSwDZDIwK8RfCtFziFA.png" /><figcaption>Example №2</figcaption></figure><p>As more tools and advancements in technology are developed, I am confident that the ability to <strong>get quick and precise answers without assistance</strong> will become even more accessible and easier to use. With these advancements, self-service BI is becoming more real than ever before, help the organization get is answers in real time, and would let data team to focus on more impactful tasks.<br>However, it is important to keep in mind that certain requirements must be met in order for self-service BI to be successful, as you can find at the next part.</p><h3>Essential Requirements for Effective Self-Service BI</h3><h4>1. Access to clean and reliable data</h4><p>As someone who has focused on data quality for the past year (for sure it’s deserves its own dedicated blog post), I can attest to its importance in any data environment. Data observability solutions such as Monte Carlo, Metaplane, Great Expectations, and Elementary (which we use at Hexa) aim to ensure that the data is of high quality by monitoring for unwanted errors. This issue is especially crucial when it comes to self-service BI, as non-technical users may be unaware of data quality issues and by that rely on incorrect information, leading to misguided decisions. In the near future, I believe that when someone asks a question, the answer will inevitably be integrated with the quality of the resource data. Thus, it is essential to establish a systematic connection between data quality and self-service BI, with data quality given the utmost importance.</p><h4>2. Well defined metrics</h4><p>It’s critical to establish well-defined metrics and make them readily available to users. This means aligning analytics engineers, data engineers, and analysts to ensure everyone is on the same page in terms of what they need to measure and what insights they hope to gain from the data. With clear metrics in place and easily accessible, users can more easily identify relevant data points, ask the right questions, and extract valuable insights from the data.</p><h4>3. Data governance &amp; permission control</h4><p>This requirement is must have for sensitive data.</p><p>Using tools like Slack bots to provide quick answers requires strict access controls to limit user access to authorized data and information. For example, HR data should be restricted to the HR group, CEO, and direct managers. Without proper controls, there is a risk of data leaks or breaches, which can be detrimental to an organization. Therefore, it’s crucial to have a robust system in place to ensure data security and compliance even though it can be quite an headache.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7f2923d110c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maximizing Your Data’s Value using Activity Schema Data Model]]></title>
            <link>https://medium.com/@baruchjacob/maximizing-your-datas-value-using-activity-schema-data-model-c796bea41c4f?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c796bea41c4f</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-transformation]]></category>
            <category><![CDATA[data-build-tool]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Wed, 18 Jan 2023 06:49:10 GMT</pubDate>
            <atom:updated>2023-02-10T06:52:13.687Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The first time I was introduced to the concept of Activity Schema was about four years ago.<br>I had the opportunity to learn about it from one of the most brilliant minds in the field of data (we referred to it as the “Layer Model”- which I still believe it’s more fitting name for this concept) Since then, I have consistently chosen to use it as my preferred schema for data modeling.<br>I find it to be the most suitable option for a wide range of use cases in modern data warehouse settings.</p><h3>About the Concept</h3><p>The core concept of this schema is to organize all data that is related to a business case into a <strong>single </strong>time series table. <br>In that way this table would be our <strong>one </strong>source of truth, and would be the only table that data team would rely on.</p><h4>Advantages</h4><ul><li><strong>Easy to maintain: </strong>if a schema change is made at the database, it would be reflected as <strong>one </strong>change in <strong>one </strong>location during the transformation process at the data warehouse.<br>This makes it simpler for our team to work with the data and keeps things running smoothly.</li></ul><p>The image below shows an example of a schema change for a user’s country</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*y2zqQo7GrOikyu19nie-dg.png" /></figure><p>An update would be made only in one place during the transformation process, preserving the integrity of all downstream processes.</p><ul><li><strong>Single source of truth: </strong>This term is often used, but not always correctly. In this schema, all of the data team outputs are based on a single table. <br>Data/Analytics engineers are responsible for ensuring quality and accuracy. <br>This ensures that this schema is making a correct and real use of this term.</li><li><strong>Time Saver: </strong>The data team can quickly and efficiently answer complex questions, leading to better decisions and insights.</li></ul><h4>Disadvantages</h4><ul><li><strong>Complexity: </strong>depending on the data complexity, creating and maintaining can be complex and time- consuming task. <br>A very precise encapsulation during the transformation process, clear documentation/ data catalog can help overcome this possible disadvantage.</li><li><strong>Limited flexibility</strong>: If the database schema is updated but the activity schema has not yet been updated. Data analysts may be querying outdated data from the activity schema.<br>A clear observability of the transformation process, and visibility of what is not updated, would help analysts figure out what data is outdated and offer them access to the updated data if needed for an analysis before the update occurs at the activity schema</li></ul><h3>Example №1</h3><p>let’s start with a simplified example that is not an actual activity schema but illustrates the concept perfectly.</p><p>The image below shows the snowflake-schema of the user’s attributes data in the database. To locate the users table, follow the green arrows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0N9EHlHntDhvZTUmjVTcw.png" /></figure><p>We aim to avoid the need for stakeholders to repeatedly perform the same joins when analyzing data (In a more realistic scenario, analysts may spend a couple of hours troubleshooting join issues) and by that allowing them to spend more time gaining insights and making an impact.</p><p>The solution for this case, at most of the times would be to give them <strong>one</strong> single table that contains all the relevant user’s attributes, by that all of them would analyzing the same <strong>one</strong> source of truth.</p><p>The lineage graph is taken from DBT showing our transformation process for this case. you can see on the right at purple table, the users table that all the stakeholders would use every time in order to investigate users data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*QvvAkOBibSzYY0j-9tfPpQ.png" /></figure><p>Here’s an example of the preferred structure of the users table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*MxHjILHpU3-FUqnFcDXPCg.png" /></figure><h3>Example №2</h3><p>The image below illustrates a real-world example of web activity modeled using an activity schema.<br>The activity table can be found under the green arrow, all the activity tables (a total of 8) can be found under the orange arrows, and the users table, from the previous example, can be found under the blue arrow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QkI9a4KR5cfMbvyHNBrC_w.png" /></figure><p>Here’s an example of the basic structure of an activity table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*s6bYhh8rxyPUV8HF_mFzWA.png" /></figure><p>The image below illustrates the relationship between the activity table (shown in purple) and the downstream tables (shown in blue) that are created for analysis and dashboard readiness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*Zl3tzJ3nPA8tetzuiQ2EyQ.png" /></figure><h3>Worth to check</h3><p>I looked for tools that use the activity schema and till now I found only one, <a href="https://www.narratordata.com/">Narratordata</a>. <br>They also provide a very deep <a href="https://github.com/ActivitySchema/ActivitySchema/blob/main/2.0.md#conceptual-overview">overview and explanation</a> of how<em> </em>to set up the schema easily and quickly. <br>If you know any other tools that use this schema, I’ll be more than happy to know.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c796bea41c4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is your favorite Bias? (AKA, avoiding biases in NPS survey)]]></title>
            <link>https://medium.com/@baruchjacob/what-is-your-favorite-bias-aka-avoiding-biases-in-nps-survey-1aa49044391a?source=rss-ba92e1017bc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1aa49044391a</guid>
            <category><![CDATA[nps-tips]]></category>
            <category><![CDATA[selection-bias]]></category>
            <category><![CDATA[measurement-bias]]></category>
            <category><![CDATA[nps-survey]]></category>
            <category><![CDATA[cognitive-bias]]></category>
            <dc:creator><![CDATA[jacob baruch]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 01:16:37 GMT</pubDate>
            <atom:updated>2023-01-13T07:10:17.946Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been intending to start writing blog posts for a long time, over this time period I wrote to myself around 15 possible topics.<br>First blog post selection was a tough decision, so I chose my favorite unbiased method and rolled the dice, however I was not happy with the results (how many blog posts about Modern Data Stack can exist)<br>So, I chose to write about one of my favorite topics — biases — and combine it with one of my least favorite — NPS surveys</p><p>As NPS surveys are frequently conducted during this time of year,<br>avoiding potential biases when analyzing their data is crucial in order to accurately determine the true NPS of an organization. <br>Without addressing these biases, the results may lead to inaccurate conclusions and ineffective actions.</p><p>Find below 4 possible biases out of many that can occur at NPS survey, and how to avoid them for more accurate results:</p><ol><li><strong>Selection Bias </strong>(sampling bias)<strong><br></strong>This occurs when the survey is sent for certain groups of users who are more likely to answer than other groups.<br><strong><em>Example:</em></strong><em> </em><br>Sending NPS surveys only to our last 3 months active users, will not give us representative view of customer satisfaction.<br><strong><em>Possible workarounds:</em></strong><br>a. Send NPS to random sample.<br>b. Monitor the results: ensure that they are representative of the overall population.</li><li><strong>Response Bias </strong>(measurement bias)<br>This occurs when certain groups of users are more likely to respond to NPS surveys than others.<br><strong><em>Example:</em></strong><br>Users who are more active may be more likely to respond than those who are less active — as a result the NPS could be higher than it’s supposed to be.<br><strong><em>Possible workarounds:</em><br></strong>a. Ensure that the survey is being sent to a representative sample of customers, including those who are less likely to respond or who likely to be neutral.<br>b. follow up with no responders: use reminder / personalized follow up etc.<br>c. use weighting techniques: (only if there is non-response bias) by giving more weight for under-represented group.<br>for example, if we know that 50% of users coming from Argentina, however, only 30% of the NPS responders came from Argentine<br>we can give more weight for the Argentinian responders<br>d. encourage response by offering incentive: Actually I’m not in favor of that as it can lead to response bias…</li><li><strong>Framing Bias </strong>(Cognitive bias)<strong><br></strong>This occurs by the way that the survey questions are worded or presented.<br><strong><em>Examples:</em></strong><em> </em><br>a. if the question is worded in a negative way — it would lead to more negative response.<br>b. using a red background or text may lead to more negative responses.<br><strong><em>Possible workarounds:</em><br></strong>a. use neutral language and colors — avoid words/colors that may have a positive or negative connotation.<br>b. avoid leading questions.</li><li><strong>Desirability Bias </strong>(Cognitive bias)<strong><br></strong>Type of cognitive bias, can occurs when survey respondents give desirable answers, rather than their true opinions. <br>This can occur when respondents feel pressure to provide positive feedback.<br><strong><em>Example:</em></strong><br>Users who are part of our customer club may feel pressure to provide positive feedback, as they do not want to be perceived as a less loyal user.<br><strong><em>Possible workarounds:</em><br></strong>a. Anonymity : allow respondents to provide anonymous feedbacks, by that they can feel more comfortable and provide more honest answers.<br>b. Provide clear instructions: how the survey will be used, and how the feedback will be used to improve our services</li></ol><p>To conclude, NPS potential biases can appear in any other surveys, however due to their significant weight in decision making, it’s very important to monitor and mitigate any potential bias, in that way we can ensure that the results are accurate and valuable.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fj1KtMamOvvQx5PfXZ8%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fj1KtMamOvvQx5PfXZ8%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fj1KtMamOvvQx5PfXZ8%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbc097a1ab42af4c9b47189cc1c3997c/href">https://medium.com/media/bbc097a1ab42af4c9b47189cc1c3997c/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1aa49044391a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>