<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carlos Jaime on Medium]]></title>
        <description><![CDATA[Stories by Carlos Jaime on Medium]]></description>
        <link>https://medium.com/@jaimemcarlosa?source=rss-42295477e7d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YPEuutzFTHoV2387onbMlA.jpeg</url>
            <title>Stories by Carlos Jaime on Medium</title>
            <link>https://medium.com/@jaimemcarlosa?source=rss-42295477e7d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:45:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jaimemcarlosa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Click to Answer: How AI is Reshaping the Economics of the Internet]]></title>
            <link>https://medium.com/@jaimemcarlosa/from-click-to-answer-how-ai-is-reshaping-the-economics-of-the-internet-65ec503a4fcc?source=rss-42295477e7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/65ec503a4fcc</guid>
            <category><![CDATA[ads]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[internet]]></category>
            <dc:creator><![CDATA[Carlos Jaime]]></dc:creator>
            <pubDate>Sun, 05 Oct 2025 20:43:57 GMT</pubDate>
            <atom:updated>2025-10-05T20:51:40.861Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>For two decades, a silent pact powered the web: content in exchange for clicks. AI just shattered that agreement, and a new battle for the future of information has begun.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5X3yVm16tm4-2ZyA" /><figcaption>Photo by <a href="https://unsplash.com/@joshuaearle?utm_source=medium&amp;utm_medium=referral">Joshua Earle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I remember, not so many years ago, those <em>caffeine-fueled </em>☕️ nights working on university papers. The mission was simple: <em>dive into Google, open an absurd number of tabs, copy paragraphs, save links, and try to weave a coherent summary</em>. In the midst of that digital odyssey, I often marveled at the miracle of having all that information for “free.” The only price was effort, the patience to get to the eleventh “o” in “Gooooooogle,” and the dexterity to dodge banner ads that flickered like neon signs in a dark alley.</p><p>Those ads, annoying at first, became part of the landscape. They were the silent toll. I understood, in my own way, that someone paid for them to be there, and that was the cost of my unlimited access. It was a fair pact, a bearable balance. Websites strived to create useful, updated content, and in return, they got my visit (and the advertisers’ cents). Leaving aside the fact that, as I later learned, the real product was my data, the cost-benefit relationship worked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*5qTws42fGUPEW7-CTth5Zw.jpeg" /><figcaption>That’s what our web browser was like back then.</figcaption></figure><p>Today, that equilibrium that sustained the web for two decades is crumbling. AI, which has infiltrated almost every aspect of our lives, has arrived in browsing with the subtlety of a bull in a china shop. For us, the users allergic to deep research and addicted to instant gratification, it’s a marvel. For content creators and the entire ecosystem that depended on our visits, it’s an existential crisis.</p><h3>🧶 The Ecosystem’s Fracture: Data from a Paradigm Shift</h3><p>The old model, the <strong>“Internet of the Click”, </strong>was based on a simple exchange: content for traffic. But we have fully entered the <strong>“Internet of the Answer”,</strong> where AI acts as an all-knowing butler, serving us information on a silver platter and eliminating the need to visit the kitchen (the original websites).</p><p>This change isn’t a theory; it’s a reality measured in cold, hard numbers. The main culprit is the phenomenon of <strong>“zero-click searches.”</strong> Why visit three blogs when AI gives you the definitive answer in a shiny box at the top of the page?</p><p>📉<strong> Let’s look at the evidence of the “crime” in more detail:</strong></p><ul><li><strong>The Great Traffic Plunge:</strong> Analyses show a general decline in organic web traffic of <strong>15% to 25%</strong>, with some projections reaching as high as 34%.2 In sectors like news and media, the bleeding exceeds 20%. A report from Digital Content Next (DCN), which includes major publishers, revealed an average year-over-year decrease of 10% in referrals from Google.</li><li><strong>The Zero-Click Black Hole:</strong> It’s estimated that <strong>60% of searches</strong> are now completed without the user clicking on any external site. Since Google launched its AI Overviews, this figure for news-related searches has jumped <strong>from 56% to nearly 69%.</strong></li><li><strong>The CTR as the Main Victim:</strong> A Pew Research Center study revealed something devastating: when an AI summary appears in the results, the probability of a user clicking on <em>any</em> link on that page is nearly halved, dropping from <strong>15%</strong> to<strong> 8%</strong>. Even worse, almost no one (only 1%) clicks on the sources.</li><li><em>within</em> the AI summary itself. An Ahrefs analysis of 300,000 keywords concluded that the presence of an AI Overview reduces the CTR for the top position by an average of 34.5%.</li></ul><p>A case that perfectly illustrates this is that of a law firm that attracted clients with articles answering specific questions like, “Is sex in a car legal in Alabama?”. Before, that search would lead a potential client to their website. Today, AI answers directly by citing the Alabama criminal code, and the firm loses the visit, the client, and the business opportunity, even though its article still appears in the search results. Visibility no longer guarantees a visit.</p><h3>🗞️ The Battlefield: Players, Reactions, and Current Cases</h3><p>Faced with this paradigm shift, the digital ecosystem has reacted with a mix of panic, ingenuity, and legal artillery.</p><p><strong>The Old Guard: The “Wooing and Suing” Strategy</strong></p><p>The major content players have adopted a dual strategy that the CEO of News Corp perfectly described as “<em>wooing and suing</em>”.</p><ul><li><strong>The Suing:</strong> On one hand, they bring out the lawyers. The most high-profile case is <em>The New York Times</em>’ lawsuit against OpenAI and Microsoft for copyright infringement.</li><li><strong>The Wooing:</strong> On the other hand, if you can’t beat them, take a check from them. OpenAI and Google are closing multi-million dollar licensing deals with giants like News Corp and Reddit to use their content legitimately.</li></ul><h3>🛡️ <strong>The New Guard: Building the New Infrastructure</strong></h3><p>While the titans battle it out, a group of clever startups is trying to solve the problem from the ground up by building the new “rules of the game” in the form of code.</p><ul><li><a href="https://tollbit.com/"><strong>Tollbit</strong></a><strong>: The Bouncer for Bots. </strong><br>Their idea is simple and brilliant: a “paywall for bots.” Imagine a bouncer at the door of your website telling AI bots, “Humans get in free, but you pay a cover charge.” Tollbit allows publishers to set variable rates: they can charge more for crawling breaking news than for archive articles, incentivizing the creation of unique content. Their system redirects unauthorized bots to a subdomain where they are informed of the requirements for access, allowing publishers to monetize each scrape on a pay-per-scrape basis. In the first quarter of 2025, it had already processed 15 million of these micro-transactions for 2,000 content producers.</li><li><a href="https://prorata.ai/"><strong>ProRata</strong></a><strong> and </strong><a href="http://gist.ai/"><strong>Gist.ai</strong></a><strong>: The VIP Content Club.</strong> <br>This startup, led by online advertising pioneer Bill Gross, proposes a different model: an answer engine, Gist.ai, that runs <em>exclusively</em> on licensed content from over 700 partner publications, such as the <em>Financial Times</em> and <em>The Atlantic</em>. Their promise is to share 50% of the ad revenue with the creators whose content helped generate the answer. They use a proprietary algorithmic process to attribute credit fairly for each use. It’s an attempt to create an ethical AI ecosystem, a “walled garden” of high-quality information.</li></ul><p>These new players aren’t just creating companies; they’re trying to build a new economic layer on top of the internet — a machine-readable protocol for the exchange of value.</p><h3>👨‍💻 Expert Analysis: Two Visions for the Future</h3><p>To understand where we’re headed, it’s helpful to contrast the views of two of the most respected tech analysts.</p><ul><li><a href="https://stratechery.com/concept/ai/implications-ai/"><strong>Ben Thompson (Stratechery)</strong></a><strong>: The Ultimate Aggregator.</strong> From the perspective of his famous “Aggregation Theory,” AI is the ultimate aggregator. It commoditizes its supply (the web’s content) and controls the user relationship. For Thompson, the only long-term survival strategy for publishers is to escape this commoditization. They must build such a strong brand and direct connection with their audience (through newsletters, apps, etc.) that they become a “destination” people seek out intentionally, rather than an anonymous source in an AI summary.</li><li><a href="https://www.ben-evans.com/benedictevans/category/Artificial+Intelligence"><strong>Benedict Evans: </strong></a><strong>The Slow Hunt for a Product.</strong> Evans offers a more cautious view. He distinguishes between the <em>technology</em> (LLMs, which are revolutionary) and the <em>product</em> (the current chatbot interface). He argues that the current way of interacting with AI is not necessarily the final form. He warns against assuming that business models and user behaviors have already solidified. For him, we are still in the “slow, boring hunt for product-market fit,” and the real disruption will unfold as this technology gets integrated into more specific tools and workflows.</li></ul><p>In short, while Thompson sees an unstoppable force demanding a radical reinvention toward direct branding, Evans reminds us that the game isn’t over yet and the final form of this technology is still being defined.</p><h3>🌎 Strategic Recommendations for a Post-Search World</h3><p>Based on this landscape, several strategic courses of action emerge.</p><p><strong>For Publishers and Media Companies:</strong></p><ol><li><strong>Diversify Aggressively:</strong> Dependence on Google search is now an existential risk. It’s imperative to invest in direct channels (newsletters, apps, communities) and in formats that AI struggles to process (long-form video, podcasts, live events).</li><li><strong>Adopt GEO Principles:</strong> Reorient content strategy to prioritize authority, structure, and verifiability. The goal is to become an indispensable source for AI models.</li><li><strong>Explore New Monetization Avenues:</strong> Actively experiment with emerging models, whether it’s integrating with platforms like Tollbit or seeking partnerships with licensed ecosystems like Gist.ai.</li></ol><p><strong>For Brands and Businesses in General:</strong></p><ol><li><strong>Strengthen the Brand:</strong> Brand recognition is the ultimate defense. If users search for you by name, they bypass the generic AI response. Investment in brand marketing is more critical than ever.</li><li><strong>Dominate a Niche with “Unhookable” Content:</strong> Become the go-to source by publishing proprietary data, detailed case studies, and expert analysis that AI cannot replicate, only cite.</li><li><strong>Fight Fire with Fire:</strong> Use AI tools to increase efficiency in your own content production and to enhance the user experience on your site with chatbots and personalization.</li></ol><h3>Conclusion: Towards a Web of Programmatic Value</h3><p>We are witnessing in real-time the transition from a web of “destinations” (sites to visit) to a web that functions as a gigantic “database” that AI agents query on our behalf. Value no longer lies just in attracting human eyeballs, but in being a reliable data source for the machine’s eyes.</p><p>The long-term challenge is existential. Creating high-quality content, like the kind I sought during my late-night study sessions, is expensive. If the economic model that supported it — that silent pact of clicks and ads — evaporates without a replacement, we risk entering a “content winter.” Experts might stop sharing their knowledge on the open web, impoverishing the very ecosystem that AIs feed on.</p><p>It would be the ultimate paradox: a technology created to access all human knowledge ends up disincentivizing the creation of new knowledge. Forging a new economic pact isn’t just a business matter; it’s a necessity to ensure the future of the internet remains a place worth exploring, even if we no longer have to get to the eleventh “o” of Google to find what we’re looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*JjHd_OKC330KafCwDD4TNg.png" /><figcaption>Meme generator.com</figcaption></figure><h3>References</h3><ul><li><strong>AI is killing the web. Can anything save it?</strong> <a href="https://www.economist.com/business/2025/07/14/ai-is-killing-the-web-can-anything-save-it">https://www.economist.com/business/2025/07/14/ai-is-killing-the-web-can-anything-save-it</a></li><li><strong>Study reveals AI reduces clicks on traditional websites by 35%…</strong> <a href="https://www.instantaneas.tic.bo/2025/08/05/tecnologia/impacto-ia-trafico-web-publicidad-digital/">https://www.instantaneas.tic.bo/2025/08/05/tecnologia/impacto-ia-trafico-web-publicidad-digital/</a></li><li><strong>September’s highlight: the silent drop in publisher site traffic.</strong> <a href="https://adrenalead.com/es/blog/lo-que-esta-de-moda-en-septiembre-la-caida-silenciosa-del-trafico-de-los-sitios-de-editores/">https://adrenalead.com/es/blog/lo-que-esta-de-moda-en-septiembre-la-caida-silenciosa-del-trafico-de-los-sitios-de-editores/</a></li><li><strong>AI marketing statistics for 2025.</strong> <a href="https://www.seo.com/es/ai/ai-seo-statistics/">https://www.seo.com/es/ai/ai-seo-statistics/</a></li><li><strong>AI-powered searches cause a decrease in web traffic.</strong> <a href="https://www.hint.mx/blog/busquedas-por-ia-causan-disminucion-de-trafico-web">https://www.hint.mx/blog/busquedas-por-ia-causan-disminucion-de-trafico-web</a></li><li><strong>Google AI Overviews linked to 25% drop in publisher referral traffic, new data shows.</strong> <a href="https://digiday.com/media/google-ai-overviews-linked-to-25-drop-in-publisher-referral-traffic-new-data-shows/">https://digiday.com/media/google-ai-overviews-linked-to-25-drop-in-publisher-referral-traffic-new-data-shows/</a></li><li><strong>The effect of AI-powered search on brands: who wins, who loses, and why.</strong> <a href="https://www.adlatina.com/publicidad/el-efecto-de-la-bsqueda-con-ia-en-las-marcas-quin-gana-quin-pierde-y-por-qu">https://www.adlatina.com/publicidad/el-efecto-de-la-bsqueda-con-ia-en-las-marcas-quin-gana-quin-pierde-y-por-qu</a></li><li><strong>The Numbers Don’t Lie: Publishers Are Losing 25% of Their Traffic to AI Platforms.</strong> <a href="https://www.google.com/search?q=https.infactory.ai/blog/the-numbers-don-t-lie-publishers-are-losing-25-of-their-traffic-to-ai-platforms">https.infactory.ai/blog/the-numbers-don-t-lie-publishers-are-losing-25-of-their-traffic-to-ai-platforms</a></li><li><strong>Google users are less likely to click on links when an AI summary appears in the results.</strong> <a href="https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/">https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/</a></li><li><strong>AI Overviews reduce clicks by 34.5%.</strong> <a href="https://ahrefs.com/blog/es/ai-overviews-reducen-clics/">https://ahrefs.com/blog/es/ai-overviews-reducen-clics/</a></li><li><strong>How can companies survive if Google’s AI takes their traffic away?</strong> <a href="https://es-us.noticias.yahoo.com/empresas-sobrevivir-ia-google-quita-193000102.html">https://es-us.noticias.yahoo.com/empresas-sobrevivir-ia-google-quita-193000102.html</a></li><li><strong>The decline of organic traffic and the rise of AI force media to seek new loyalty channels.</strong> <a href="https://laboratoriodeperiodismo.org/el-declive-del-trafico-organico-y-el-auge-de-la-ia-obligan-a-los-medios-a-buscar-nuevas-vias-de-fidelizacion/">https://laboratoriodeperiodismo.org/el-declive-del-trafico-organico-y-el-auge-de-la-ia-obligan-a-los-medios-a-buscar-nuevas-vias-de-fidelizacion/</a></li><li><strong>HUMAN and TollBit: The dynamic duo to mitigate, control, and monetize AI scraping agents.</strong> <a href="https://www.humansecurity.com/learn/blog/human-and-tollbit-the-dynamic-duo-to-mitigate-control-and-monetize-ai-scraping-agents/">https://www.humansecurity.com/learn/blog/human-and-tollbit-the-dynamic-duo-to-mitigate-control-and-monetize-ai-scraping-agents/</a></li><li><strong>Stratechery by Ben Thompson — The Implications of AI.</strong> <a href="https://stratechery.com/concept/ai/implications-ai/">https://stratechery.com/concept/ai/implications-ai/</a></li><li><strong>Benedict Evans — Essays on Artificial Intelligence.</strong> <a href="https://www.ben-evans.com/benedictevans/category/Artificial+Intelligence">https://www.ben-evans.com/benedictevans/category/Artificial+Intelligence</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65ec503a4fcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling the Magic Behind AI Text: A Deep Dive into LLM Sampling Techniques]]></title>
            <link>https://medium.com/@jaimemcarlosa/unveiling-the-magic-behind-ai-text-a-deep-dive-into-llm-sampling-techniques-3eaa9ace3f06?source=rss-42295477e7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eaa9ace3f06</guid>
            <dc:creator><![CDATA[Carlos Jaime]]></dc:creator>
            <pubDate>Mon, 05 May 2025 03:08:27 GMT</pubDate>
            <atom:updated>2025-05-05T03:15:56.830Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Go beyond the prompt — learn how Temperature, Top-K, and Top-P shape AI creativity, coherence, and everything in between.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYs9jlCRQ15EVuzC" /><figcaption>Photo by <a href="https://unsplash.com/@glencarrie?utm_source=medium&amp;utm_medium=referral">Glen Carrie</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>🤖 We interact with <em>Large Language Models</em> (LLMs) like Gemini, ChatGPT, Claude, and others daily. We give them prompts, and they generate text — sometimes factual, sometimes creative, occasionally bizarre. But have you ever stopped to think about the precise mechanism that decides <em>which word </em>comes next? It’s not just magic; it’s a fascinating interplay of probabilities and clever selection methods called <strong>sampling techniques</strong>.</p><p>Understanding these techniques is like finding the hidden control panel for your AI. It moves you from simply <em>using</em> LLMs to strategically directing them. Let’s pull back the curtain.</p><h3>What’s Really Happening When an LLM Writes?</h3><p>Contrary to intuition, an <em>LLM doesn’t predict one single perfect word to follow your prompt</em>. Instead, based on its training data and the input sequence, it calculates a probability score for <em>every single possible word</em> (or “token,” a piece of a word) in its vast vocabulary.</p><blockquote>Imagine asking an LLM to complete 🌤 “<strong>The sky is…</strong>” It might assign probabilities like:</blockquote><blockquote><strong>blue</strong>: 65% <br><strong>gray</strong>: 15%<br><strong>dark</strong>: 10%<br><strong>falling</strong>: 0.01%<br><strong>banana</strong>: 0.00001%<br>…and so on for thousands of other tokens.</blockquote><p>The LLM now has a ranked list of possibilities. But how does it choose just one to continue the sentence? That’s where sampling comes in.</p><h3>The Crucial Role of Sampling</h3><p>Sampling techniques are the algorithms used to select the next token from that calculated probability distribution. They are the gatekeepers determining the final output.</p><blockquote><em>Why are they so important? Because they directly control the fundamental characteristics of the generated text</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*es-AKrKD1DdWy2ZmCIRKQg.png" /></figure><blockquote>Mastering these techniques involves tweaking specific parameters, the most common of which are <em>Temperature</em>, <em>Top-K</em>, and <em>Top-P.</em></blockquote><h3>Decoding the Controls: Temperature, Top-K, Top-P</h3><p>Think of these as knobs on your AI’s control panel. Let’s explore each one with examples. Imagine we give an LLM the simple prompt:<em> “Write a short sentence about a cat.”</em></p><h3>1. Temperature: The Creativity Thermostat 🔥❄️</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RmHZRqr4c0QT30BHbDm3gg.png" /></figure><h4>How it works:</h4><ul><li><strong>Low Temperature </strong>❄️<strong> (e.g., 0.1–0.4): </strong>It sharply increases the probability of the most likely tokens and diminishes the chances of less likely ones. This leads to focused, coherent, and often repetitive or predictable output. A temperature of 0 is “greedy decoding” — it always picks the single most probable token.</li><li><strong>High Temperature </strong>🔥<strong> (e.g., 0.8–1.2+): </strong>It flattens the distribution, making probabilities more even. Less likely tokens get a higher chance of being selected. This results in more diverse, creative, surprising, and potentially less coherent or random output.</li></ul><blockquote><strong>Example Scenario: <br>Prompt: </strong>“Write a short sentence about a cat.”</blockquote><blockquote><strong>Temp = 0.1: <em>“</em></strong><em>The cat sits on the mat.“</em> <em>(Very standard, highest probability choice)</em><br><strong>Temp = 0.7:</strong> <em>“</em>A fluffy white cat sleeps peacefully in the sunbeam.<em>”</em> <em>(Still sensible, but more descriptive)</em><br><strong>Temp = 1.2:</strong> “Cats, those enigmatic void-rulers, often ponder quantum states between naps.”<em> (</em><strong><em>Unexpected</em></strong><em>, potentially off-topic, highly creative or slightly nonsensical, Banana!)</em></blockquote><h4>💡 When to Use:</h4><ul><li><strong>Low:</strong> Factual summarization, data extraction, code generation, Q&amp;A where precision is key.</li><li><strong>High: </strong>Brainstorming, story writing, poetry generation, creating varied options.</li></ul><h3>2. Top-K: Limiting the Candidate Pool 🎯</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u7oPgFvoI_kTHX0cKDMfJQ.png" /></figure><h4>How it works:</h4><p>Before sampling (often influenced by <em>temperature</em>), the LLM identifies the <em>K tokens with the highest probabilities</em>. All other tokens are discarded (probability set to zero) for this step. The next token is then chosen only from this reduced set.</p><ul><li><strong>Low K (e.g., K=1, K=5): </strong>Severely limits the choices. K=1 is equivalent to greedy decoding (<em>same as Temp=0</em>). Leads to more focused and predictable text.</li><li><strong>High K (e.g., K=50, K=100):</strong> Allows a wider range of tokens to be considered, promoting more diversity while still filtering out the extreme long-tail options.</li></ul><blockquote><strong>Example Scenario </strong><br><strong>Prompt: </strong>“Write a short sentence about a cat.”</blockquote><blockquote><strong>K = 3: </strong>Let’s say the top 3 next words after “A cat…” are “is”, “sleeps”, “plays”. The LLM must choose one of these. <strong>Output</strong>: A cat sleeps.</blockquote><blockquote><strong>K = 20:</strong> The pool now includes “is”, “sleeps”, “plays”, “dreams”, “hunts”, “watches”, “purrs”, etc. Output could be: A cat watches the birds outside.</blockquote><h4>💡 When to Use:</h4><ul><li><strong>Low</strong>: When you want to prevent the model from going too far off-piste, ensuring relevance even with higher temperatures.</li><li><strong>High</strong>: When you want diversity but need to avoid completely nonsensical words that might appear with very high temperatures alone.</li></ul><h3>Top-P (Nucleus Sampling): Dynamic Diversity 💡</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMzAPAa3YdYa6RLrFwSTWw.png" /></figure><h4><strong>How it works</strong></h4><p>Tokens are ranked by probability. The LLM sums their probabilities starting from the highest until the sum reaches or exceeds P. Only the tokens included in this sum are considered for sampling.</p><ul><li><strong>Low P (e.g., P=0.1, P=0.5):</strong> Often includes only one or a very few highly probable tokens. Similar effect to low K or low temperature — <strong>focused output</strong>.</li><li><strong>High P (e.g., P=0.9, P=0.95):</strong> Includes a wider range of tokens, allowing for <strong>more diversity</strong>. P=1 considers all tokens with non-zero probability.</li><li><strong>The Key Advantage:</strong> Top-P is <em>dynamic</em>.</li><li>If the model is very confident about the next word (e.g., after “New York…”, “City” might have 90% probability), a high P (like 0.92) might <em>still</em> only include “City” in the nucleus.</li><li>If the model is uncertain (e.g., after “The cat felt…”, probabilities for “happy”, “sleepy”, “curious”, “hungry” might be more evenly spread), the <em>same</em> P value (0.92) would include a larger set of tokens to reach the threshold.</li></ul><blockquote><strong>Example Scenario <br>Prompt: </strong><em>“Write a short sentence about a cat.”</em></blockquote><blockquote><strong>P = 0.5:</strong> If “The” (40%), “A” (30%), “My” (10%) are top probabilities after the start, the nucleus might just be {“The”, “A”} (cumulative 70%). Output starts with “A cat…”.</blockquote><blockquote><strong>P = 0.95:</strong> The nucleus might expand to include {“The”, “A”, “My”, “That”, “One”…} until 95% probability is covered. This allows for more varied sentence beginnings. Later, if the model is very sure the next word is “paws”, P=0.95 might <em>only</em> select “paws”.</blockquote><h4>💡 <strong>When to Use:</strong></h4><p>Often preferred over Top-K because it adapts the pool size based on the model’s confidence at each step, providing a good balance between coherence and diversity.</p><h3>Why This Matters For You</h3><p>Whether you’re a developer integrating LLMs, a writer using AI assistance, a researcher studying model behavior, or just a curious user, understanding sampling gives you power:</p><ul><li><strong>Fine-tune Outputs:</strong> Tailor AI responses precisely to your task. Need a factual report? Lower the temperature. Need creative marketing slogans? Crank it up.</li><li><strong>Troubleshoot Issues:</strong> Is the AI too repetitive? Maybe Temp/K/P are too low. Is it nonsensical? Maybe Temp is too high.</li><li><strong>Become a Better Prompt Engineer:</strong> Knowing these controls complements crafting effective prompts, leading to superior results.</li></ul><h3>How They Work Together</h3><p>Often, these settings are used in combination. A common scenario in platforms like Vertex AI Studio is that tokens must pass <em>both</em> Top-K and Top-P filters. Then, Temperature is applied to sample from the remaining candidates. Extreme settings in one can override others (e.g., Temp=0 or K=1 makes others irrelevant). Finding the right balance often requires experimentation.</p><h3><strong>Let’s use a simple and open-ended prompt to see how responses change with different settings.</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAYmBoz2c4HJwKUNSHr8yw.png" /></figure><p><strong>Base Prompt:</strong></p><pre>Write a sentence that continues this idea: <br>The old lighthouse stood solitary on the cliff, and every night…</pre><h4><strong>Example 1:</strong> Balanced Configuration (Coherent but with some creativity)</h4><p><strong>Temperature</strong>: 0.5 | <strong>Top-K:</strong> 40 | <strong>Top-P</strong>: 0.95</p><blockquote><strong>Expected Effect:</strong> A logical and common continuation, but not necessarily the only or most obvious one. It might have a slightly descriptive or poetic touch.</blockquote><blockquote><strong>Illustrative Result:</strong><em> “…</em>The old lighthouse stood solitary on the cliff, and every night, its faint beam cut through the darkness, guiding lost sailors safely home.<em>”</em></blockquote><blockquote><strong>Comment:</strong> This is a very standard and expected continuation for a lighthouse. Coherent and descriptive.</blockquote><h4><strong>Example 2: Low Temperature (Deterministic / Focused)</strong></h4><p><strong>Temperature</strong>: 0.1 | <strong>Top-K:</strong> 40 | <strong>Top-P</strong>: 0.95</p><blockquote><strong>Expected Effect:</strong> The most probable and common response, very predictable. Little or no creative deviation.</blockquote><blockquote><strong>Illustrative Result:</strong> “…its beam of light pierced the darkness, guiding the ships.”</blockquote><blockquote><strong>Comment:</strong> Very direct and to the point. It focuses on the main and most obvious function of the lighthouse, eliminating almost any creative element. It’s the “safest” answer.</blockquote><h4><strong>Example 3: High Temperature (Creative / Random)</strong></h4><p><strong>Temperature</strong>: 1.0 | <strong>Top-K</strong>: 40 | <strong>Top-P</strong>: 0.95</p><blockquote><strong>Expected Effect:</strong> A more unexpected, creative response, perhaps a bit strange or less logically connected. It introduces less common elements.</blockquote><blockquote><strong>Illustrative Result:</strong> “…it told silent stories to the wind about ancient shipwrecks and ghosts of fishermen who were still searching for the shore.”</blockquote><blockquote><strong>Comment:</strong> It introduces imaginative elements (silent stories, ghosts) that are not the literal function of the lighthouse. It’s more poetic and less predictable. With even higher temperatures, it could become incoherent.</blockquote><h4><strong>Example 4: Low Top-K (Restricted Vocabulary)</strong></h4><p><strong>Temperature</strong>: 0.7 (moderate to allow some variety) | <strong>Top-K</strong>: 5 |<strong> Top-P</strong>: 0.95 (high, but limited by Top-K)</p><blockquote><strong>Expected Effect:</strong> The model can only choose from the 5 most probable words at each step. This can lead to simpler or more repetitive wording if the main options are limited.</blockquote><blockquote><strong>Illustrative Result:</strong> “…the strong light shone over the dark sea, always the same light.”</blockquote><blockquote><strong>Comment:</strong> The word choice is perhaps a bit simpler (‘strong light’, ‘dark sea’). The K limitation can force the model to use more generic words if the more specific options fall outside the top 5.</blockquote><h4><strong>Example 5: Low Top-P (Very Small Nucleus / Focused)</strong></h4><p><strong>Temperature</strong>: 0.7 (moderate) | <strong>Top-K</strong>: 40 (high, but limited by Top-P) |<strong> Top-P</strong>: 0.5</p><blockquote><strong>Expected Effect:</strong> Similar to a low temperature, but dynamic. Only the most probable tokens that sum up to 50% of the total probability are considered. Forces the model to choose high-confidence options.</blockquote><blockquote><strong>Illustrative Result:</strong> “…it sent its light signal across the waves, warning of nearby danger.”</blockquote><blockquote><strong>Comment:</strong> It focuses on an essential and probable function of the lighthouse (signal, warning). The low P value trims the less probable options, resulting in a very focused and less creative continuation, similar to lowering the temperature.</blockquote><h3>Conclusion: Master the Knobs, Master the Output</h3><p>LLM sampling techniques (<em>Temperature</em>, <em>Top-K</em>, <em>Top-P</em>) are the hidden gears driving the text generation process. They bridge the gap between probabilistic calculations and the final words you see on the screen. By learning to adjust these “knobs,” you gain significant control over the AI’s output, enabling you to harness its power more effectively for coherence, creativity, or the perfect blend of both.</p><p>So next time you’re configuring an LLM interaction, don’t ignore those sampling settings. Experiment, observe, and unlock a new level of control over your AI collaborators!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eaa9ace3f06" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So, You Wanna Talk Data? Why Data Lineage is Your New BFF]]></title>
            <link>https://medium.com/@jaimemcarlosa/so-you-wanna-talk-data-why-data-lineage-is-your-new-bff-925e6fd1542e?source=rss-42295477e7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/925e6fd1542e</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[lineage]]></category>
            <dc:creator><![CDATA[Carlos Jaime]]></dc:creator>
            <pubDate>Wed, 12 Jun 2024 03:14:39 GMT</pubDate>
            <atom:updated>2024-06-12T03:18:53.628Z</atom:updated>
            <content:encoded><![CDATA[<h4>More than a co-worker, he is a friend… a good friend who will make our lives easier… (or so I hope)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2HxO04mhdQo_wWig" /><figcaption>Photo by <a href="https://unsplash.com/@danroizer?utm_source=medium&amp;utm_medium=referral">Dan Roizer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Hello Data Lover! 🧑‍💻</h4><p>Undoubtedly, one of the most rewarding experiences during my childhood camping years was the time when I would sit and watch a river flow down the mountain 🏕. The whisper of the water would envelop me as I watched it meander down the mountainside. That spectacle gave me an incomparable feeling of peace and serenity.</p><p>Imagine for a moment that river descending from the top of the mountain. Its waters, no doubt, have an origin; perhaps they spring from a serene lake, flow from a majestic glacier or emerge from the depths of some mysterious cave. As it advances, the river encounters obstacles: waterfalls that defy its course, tributaries that add to its flow. Perhaps it flows into smaller lakes before reaching its final destination, whether it is a large gentle lake or the vast ocean.</p><p>Now, let’s translate this analogy to the <strong>world of data.</strong> Think of each drop of water in that river as if it were a datum, with a definite origin and undergoing various transformations along its journey. It joins with other rivers, i.e., data from other sources, and continues its flow, undergoing a series of changes until it reaches its final destination, where it can be tapped.</p><p>📌 This process of understanding and defining the flow and its multiple stages and transformations from the starting point to the destination of the data, is what we know as the “<em>lineage</em>” of data. More technically, we could define it as the process of tracing, documenting and visualizing the journey of data from its origin to its final use. This process allows us to identify the complete flow of data, including all the transformations that the information undergoes along the way, as well as the places where it is stored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAQOH66F3k9LVJ9z8-9B9A.png" /><figcaption>Example of basic data lineage</figcaption></figure><p>Lineage not only involves documenting each stage or node present in the flow, but also generating mechanisms so that it can be transmitted between different teams throughout the organization, such as the generation of a visual component that identifies each element, and is nourished with different metadata that generates value and reinforces the objective of the data lineage.</p><p>There are several ways to categorize the data lineage, each offering a different view of the data journey:</p><h4>📝<strong> By documentation method:</strong></h4><ul><li><strong>Descriptive data lineage: </strong>This is the old-school approach, where the data flow is documented manually. Imagine meticulously drawing the path of the river on a paper map, keeping note of each source, transformation, and destination. While it provides basic insight, it is time-consuming, error-prone and quickly becomes unwieldy with complex data pipelines.</li><li><strong>Automated data lineage: </strong>Here, technology is of great help. Tools are used to automatically capture and track the flow of data. Imagine using satellite photos, which draw you the path of the river and which we can automate to constantly update any changes. Using automated tools, it is certainly a much more efficient and accurate way to understand the flow of data.</li></ul><h4>📐 <strong>By lineage technique:</strong></h4><ul><li><strong>Design Lineage: </strong>This focuses on the architect’s blueprint, the intended data flow as conceptualized in the system design. This blueprint specifies each process and transformation through which the data will pass. It provides a general understanding of how data should move within the system.<br>For example, when you start a project for modeling a new data flow from a new source.</li><li><strong>Business Lineage: </strong>This perspective turns the storyline on its head, focusing on the “why” of the data flow. It explains how data contributes to specific business processes. Business lineage connects the data flow to its eventual impact on business objectives.<br>For example, identifying the main elements that make up the finance data flow, from the source which may be a backend API, to the dashboard with the main finance KPIs.</li><li><strong>Operational lineage: </strong>This dives into the details, detailing the actual movement of data through the system. Operational lineage reveals the technical transformations applied to the data at each stage.<br>It depends on the metadata in each element of the data flow, which can define technical aspects, aggregations applied, types of transformations, among others.</li></ul><h4>🙋‍♀️ <strong>By user role:</strong></h4><ul><li><strong>Business Data Lineage: </strong>This is aimed at the non-technical audience. It presents a high-level overview of data flow, focusing on how it impacts business processes. The business lineage seeks clear communication of the role of data in the broader business landscape.</li><li><strong>Technical/Design Data Lineage: </strong>This is aimed at data engineers and analysts who build and manage data pipelines. It provides a detailed view of technical transformations and data movement within the system. The technical lineage enables data professionals to understand and troubleshoot any problems that may arise.<br>Data quality checks, periodicity of execution, functional and technical owners, and other relevant metadata can be identified to aid in the rapid identification and status of data at a specific point in time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rdhm0OYrwJaPKz4TFhXzuQ.png" /><figcaption>Example of Data Lineage generated by DBT (DBT Labs)</figcaption></figure><h3>So.. what is the main advantage of using Data lineage?</h3><h4>💎 Data quality: beyond detection</h4><p>While detecting data problems is critical, it is not enough. Data Lineage allows you to go further:</p><ul><li><strong>Identify the source of inconsistencies:</strong> Discover where that bad data is coming from and eliminate the root of the problem.</li><li><strong>Implement accurate and preventive solutions: </strong>Develop effective strategies to prevent data problems from recurring.</li><li><strong>Improve problem-solving efficiency: </strong>Save time and resources by quickly identifying the source of failures.</li><li><strong>Strengthen confidence in data: </strong>Make confident decisions based on clean and reliable information.</li></ul><h4>🧭 Constantly changing data: Tame the chaos with Data Lineage!</h4><p>In the world of Big Data, where data flows like a mighty river, data sources are constantly changing — it seems like a mess! But don’t worry, Data Lineage comes to the rescue to keep you in control:</p><ul><li><strong>Where is this data coming from? </strong>With Data Lineage, you’ll know exactly where each piece of data comes from, even if it comes from multiple sources.</li><li><strong>How were these models created? </strong>Trace the origin of your data models and understand how they have transformed over time.</li><li><strong>Change of course!</strong> Data sources change, yes, but you don’t have to worry. Data Lineage adapts to those changes and keeps your models up to date.</li><li><strong>Always reliable data models.</strong> Make sure your data models are based on accurate and reliable information, even when sources change.</li></ul><p>In short, Data Lineage helps you navigate the ever-changing world of Big Data with peace of mind — invest in it and say goodbye to chaos!</p><p>Remember:<em> Data Lineage is not just a technology, it’s your compass for navigating the ever-evolving sea of data.</em></p><h4>👮🏻‍♂️ Privacy and security: granular control of data access</h4><p>In the digital age, data privacy and security are more important than ever. Data Lineage gives you the power to take full control of your data:</p><ul><li><strong>Who accesses what? </strong>Seamlessly visualize how your data flows and who has access to it, from individual users to different roles.</li><li><strong>Customized access controls. </strong>Define granular, customized access rules so that each user only has access to the data they need.</li><li><strong>Keep risks at line. </strong>Proactively identify and mitigate privacy and security risks to protect your most valuable data.</li><li><strong>Stress-free compliance. </strong>Comply with data regulations like GDPR and CCPA with ease thanks to the visibility Data Lineage gives you.</li></ul><p>In short, Data Lineage is your gatekeeper for flawless privacy and security of your data — invest in it and worry free!</p><h4>⚙️ Process control: visibility and automation</h4><p>Data lineage enriched with metadata, such as tags, documentation and owners, brings greater transparency to processes, even for non-technical users. A data lineage diagram that includes metadata about the current state of the data (last load, issues, periodicity, etc.) provides a complete and simple view of the current state of data models, including their downstream and upstream dependencies.<br>Tools such as DBT take advantage of this functionality to run models based on their dependencies. If a data model needs to be reprocessed, both its downstream and upstream dependencies can be automatically identified and processed, avoiding data loss between models.</p><ul><li><strong>Overview of your processes. </strong>See clearly how your data flows through the different processes, understanding each step and their interrelationships.</li><li><strong>Solve problems in a flash.</strong> Identify and solve problems quickly and efficiently with the visibility that Data Lineage gives you.</li><li><strong>Automated data models.</strong> Automate the execution of your data models to save time and improve efficiency.</li><li><strong>Data always safe.</strong> Reduce the risk of data loss by having complete traceability of your data journey.</li><li><strong>Barrier-free teamwork.</strong> Facilitates collaboration between technical and non-technical teams, as everyone has access to the same information about data processes.</li></ul><h4>⚡️ Enhancing the DRY principle: a unique approach to data transformation</h4><p>The DRY (Don’t Repeat Yourself) principle is fundamental for efficient and maintainable software development. In the context of data analysis, data lineage plays a crucial role in the implementation of this principle.</p><ul><li><strong>Duplication detection: </strong>A data lineage solution can easily identify data models created with the same logic and dependencies. This allows data teams to consolidate these transformations in a single place, eliminating redundancy and improving efficiency.</li><li><strong>Single source of truth: </strong>Data Lineage helps establish a single source of truth for transformed data. This makes it easier for data analysts and data engineers to work with the same information, reducing confusion and errors.</li><li><strong>Single visual language: </strong>The data lineage provides a visual representation of data transformations, making it easier for data teams to understand and communicate with each other. This allows data analysts to easily change dashboards, scripts and ML models that reference the single source of truth, while data engineers can migrate pipelines, standardize and optimize data transformations more efficiently.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*4cr4OIOxwCPJ4ZtxpE080w.gif" /><figcaption>DRY (Don’t Repeat Yourself)</figcaption></figure><h4>💸 Platform optimization: cost reduction</h4><p>The data lineage, enriched with metadata on periodicity and data volume, can be an invaluable tool for optimizing data platforms and reducing costs.</p><ul><li><strong>Identification of bottlenecks:</strong> By analyzing the data flow and the number of downstream nodes in each data model, bottlenecks in the pipelines can be easily identified. This allows data teams to focus their optimization efforts on the most critical points.</li><li><strong>Data model optimization: </strong>A deep analysis of data models with the most downstream nodes can reveal opportunities to improve their performance. This may include:<br><em>- Changing the materialization of the model.<br>- Improving the script or SQL query.<br>- Applying the DRY principle.<br>- Adjusting the update periodicity.</em></li></ul><p><strong>Benefits of data lineage for platform optimization:</strong></p><ul><li><em>Reduces infrastructure and processing costs.</em></li><li><em>Improves the performance of data pipelines.</em></li><li><em>Increases the overall efficiency of the data platform.</em></li><li><em>Frees resources for more strategic projects.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*q2_vPi94JmLeJ51tlvWZgQ.png" /><figcaption>Now it seems crazy right?</figcaption></figure><h4>🚀 Getting into Data Provenance: Beyond the Lineage</h4><p>Data lineage, as we have already seen, provides us with a detailed map of the path that data takes within a system. It is a fundamental tool for understanding how data is created, transformed and used. However, data lineage focuses primarily on the “what” and the “how”, i.e., what data is moved and how it is transformed.<br><strong>Data provenance</strong>, on the other hand, goes a step further and delves into the “who”, “when” and “why”. In other words, data provenance provides us with additional information about:</p><ul><li><strong>Origin:</strong> where did the data come from in the first place — was it internally generated, collected from external sources, or acquired through third parties?</li><li><strong>Ownership: </strong>Who owns the data at each stage of its journey? Are there different stakeholders or parties responsible for managing the data?</li><li><strong>History:</strong> What modifications have been made to the data throughout its life cycle; have filters, aggregations or transformations been applied?</li><li><strong>Context: </strong>What is the purpose of collecting and using this data? What business questions or problems are you seeking to answer?</li></ul><p>By understanding the provenance of the data, we gain a more complete and holistic view of the information we handle. It then allows us to Improve data quality, Comply with regulations, Make more informed decisions, and Increase transparency and trust among other things.<br>In short, data provenance complements data lineage by providing a deeper and more contextualized view of information. It is an invaluable tool for organizations looking to make the most of their data and ensure its responsible, ethical and secure use.</p><h3>In conclusion</h3><p>In short, data lineage is not just a technical tool, but a data management philosophy that promotes transparency, accountability and intelligent use of information. By adopting data lineage, organizations can navigate with greater security and confidence in the sea of data, transforming this asset into a source of value and knowledge for the achievement of their strategic objectives.</p><blockquote>Remember: Data lineage is not a destination, it is a continuous journey of improvement and learning. It is the compass that guides organizations into a future where data becomes a strategic ally for success.</blockquote><p>💙 <em>Thanks for reading me! see you soon</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=925e6fd1542e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The challenges and learnings of a beginner in the world of DBT (Data Build Tool)]]></title>
            <link>https://medium.com/@jaimemcarlosa/the-challenges-and-learnings-of-a-beginner-in-the-world-of-dbt-data-build-tool-58241429d956?source=rss-42295477e7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/58241429d956</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Carlos Jaime]]></dc:creator>
            <pubDate>Wed, 05 Jun 2024 02:46:04 GMT</pubDate>
            <atom:updated>2024-06-05T02:46:04.065Z</atom:updated>
            <content:encoded><![CDATA[<h3>The challenges and learnings of a beginner in the world of DBT</h3><h4>I share a brief summary of my first experiences in learning and using this beautiful tool!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sh2Zt4UUU0Bg-OH3" /><figcaption>Photo by <a href="https://unsplash.com/@foxxmd?utm_source=medium&amp;utm_medium=referral">Matt Foxx</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Hello <strong>Data Lover</strong>! 🧑‍💻</h4><p>A few months ago, I made the courageous decision to make a career change, switching jobs and focus. Yes, I’m still in the exciting world of data, but this time from the perspective of an <a href="https://medium.com/@ericfflynn/what-is-an-analytics-engineer-5701ec6e863e"><em>Analytics Engineer</em></a>. The reason? Curiosity was driving me to understand the business better, not just stay on the technical side. I wanted to go further, to collaborate with business analysts and generate a real impact on the company.</p><p>In this new path, I have embarked on learning new tools and methodologies, and among them, one that has especially captivated me: <a href="https://www.getdbt.com/"><em>DBT (Data Build Tool)</em></a> 🧡. Why? Simple: this tool allows me to develop complete data models using only SQL. Yes, as you read it! <em>No more reinventing the wheel with Python code or wasting hours configuring and managing pipelines.</em></p><figure><img alt="Data Build Tool" src="https://cdn-images-1.medium.com/max/693/1*s02LMPUx8fyD1xBscQieJg.jpeg" /></figure><p>🚀 <strong>DBT has transformed my work in many ways:</strong></p><ul><li><strong>Transform data into insights in the blink of an eye:</strong> I use SQL, a widely used language in the field of data analysis, to create complete data models in record time.</li><li><strong>Setting up data quality checks: </strong>The tool allows you to easily set up quality checks, ensuring that my data is always clean and reliable.</li><li><strong>Sherlock Holmes-like data lineage: </strong>I can trace the origin and transformations of my data as if I were a detective, even at the column level! A dream come true for any AE 🕵️‍♂️.</li><li><strong>Seamless collaboration with other analysts: </strong>DBT centralizes the whole process, facilitating collaboration and teamwork.</li><li><strong>Great community: </strong>DBT as a framework has a very active community, accessible documentation and hundreds of forums where you can get help and examples.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZQaz6UqdBWnkUMKTXTQew.png" /><figcaption>Illustration by <a href="https://undraw.co/">https://undraw.co/</a></figcaption></figure><p><em>Not sure where to start? I share with you what until today has been my initial guide to get into this world:</em></p><h3>💡 It starts with the fundamentals</h3><p>In the information age, being a technology enthusiast means navigating a sea of data. And that’s how I came to DBT, interested in how it works, terminology and versions. Google became my compass, guiding me through articles, videos and blogs like Medium. Gradually, I was putting the puzzle together, understanding the basic concepts and placing DBT within the data ecosystem.</p><ul><li><a href="https://www.getdbt.com/product/what-is-dbt">What is dbt? | dbt Labs</a></li><li><a href="https://www.y42.com/blog/dimensional-modeling">Dimensional Data Modeling with dbt (hands-on) | Y42</a></li><li><a href="https://towardsdatascience.com/what-is-dbt-data-build-tool-and-when-should-you-use-it-9dc1566d960d">What is dbt (data build tool) and When should you use it?</a></li></ul><h3>✋ Don’t skip the basics!</h3><p>Forget about <em>“I already know this, this too…”</em>, the key here is perseverance:<strong><em> Don’t skip the basics!</em></strong></p><p>It is true that experience gives us confidence, but sometimes it can lead us to underestimate the basics. In the case of DBT, if you don’t know the tool well, it is crucial to advance step by step. Each aspect is important and forms part of a solid knowledge base that will allow you to create new things and tie things together at all levels. A clear example:<em> SQL optimization.</em></p><p><em>If you already have experience, you might think you don’t need it</em>. However, the SQL optimization module for DBT opened my eyes to new perspectives and gave me valuable tools to optimize queries in BigQuery, Snowflake and, of course, DBT. Optimizing a query in a platform is not the same as optimizing it in DBT (DBT is much better!).</p><p>📌 <strong>DBT-Learn has very important on-demand courses:</strong></p><ul><li><a href="https://learn.getdbt.com/courses/dbt-fundamentals">DBT Fundamentals</a></li><li><a href="https://learn.getdbt.com/courses/refactoring-sql-for-modularity">Refactoring SQL for modularity</a></li><li>A<a href="https://learn.getdbt.com/courses/advanced-materializations">dvanced materializations</a></li><li>A<a href="https://learn.getdbt.com/courses/advanced-testing">dvanced testing</a></li><li>A<a href="https://learn.getdbt.com/courses/analyses-and-seeds">nalyses-and-seeds</a></li></ul><p><a href="https://learn.getdbt.com/catalog">dbt Learn</a></p><h3>🏅 Practice makes perfect: Let’s put what we have learned to work!</h3><p>Theory is essential, but practice takes it to another level. Over time, theory transforms into experience, allowing you to make better decisions, solve problems more fluently and exploit DBT’s functionalities to the fullest.<br><em>DBT is much more than Pipelines too!</em></p><p>I have also learned that you can generate other integrations, are you an Airflow enthusiast? You can orchestrate dbt executions too? want to use it with other data governance tools? DBT also has packages to use!</p><p><a href="https://rasiksuhail.medium.com/orchestrating-dbt-with-airflow-a-step-by-step-guide-to-automating-data-pipelines-part-i-7a6db8ebc974">Orchestrating dbt with Airflow: A Step by Step Guide to Automating Data Pipelines — Part I</a></p><h3>In conclusion</h3><ul><li>☝<strong> Don’t underestimate the foundation: </strong>Every step, no matter how simple it may seem, is crucial to building a solid foundation of knowledge.</li><li>📖 <strong>Experience is valuable, but don’t be complacent:</strong> There is always something new to learn, even in subjects you have mastered.</li><li>🌎 <strong>DBT offers you a world of possibilities:</strong> Explore its functionalities, experiment and develop your ability to solve problems creatively and efficiently.</li></ul><p><strong>DBT never ceases to surprise me every day.</strong> Every day I discover new possibilities and functionalities that help me to optimize my work as an Analytics Engineer. Undoubtedly, I still have a lot to learn along the way, but I am sure I will share with you my findings, challenges and achievements along the way.</p><blockquote>👨‍💻 About me:</blockquote><blockquote>Hello everyone! I’m a passionate reader of articles on Medium and, finally, I’ve decided to take the leap and share my own knowledge with you.</blockquote><blockquote>This is my first article, I introduce myself and tell you about my great interest in the world of technology and my love for the world of data. Throughout my career, I have gained experience in several tools and platforms, such as dbt, Snowflake and BigQuery, python, airflow among others, and I am excited to share what I have learned with you.</blockquote><blockquote>Thanks for reading me, I look forward to your feedbacks! ⭐</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58241429d956" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>