<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by varun patil on Medium]]></title>
        <description><![CDATA[Stories by varun patil on Medium]]></description>
        <link>https://medium.com/@varun.patil578?source=rss-1c5d61b74dbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3Y1fGm3pX8ZjPGiQ.jpg</url>
            <title>Stories by varun patil on Medium</title>
            <link>https://medium.com/@varun.patil578?source=rss-1c5d61b74dbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:38:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@varun.patil578/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Memento, But Make It Code: How Temporal Builds AI Agents That Never Forget]]></title>
            <link>https://medium.com/@varun.patil578/memento-but-make-it-code-how-temporal-builds-ai-agents-that-never-forget-2767cefac46f?source=rss-1c5d61b74dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/2767cefac46f</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[temporal]]></category>
            <dc:creator><![CDATA[varun patil]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 10:19:04 GMT</pubDate>
            <atom:updated>2026-01-08T10:19:04.562Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Temporal Actually Works: The Memento Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*mWHGmlOS8B2iecwJBBcA7g.jpeg" /></figure><p>The magic trick Temporal uses to “pause” for days without consuming memory is called <strong>Replay</strong>. If you’ve seen Christopher Nolan’s <em>Memento</em>, you already understand it.</p><p>Leonard Shelby can’t form new memories. Every few minutes, he forgets everything. Yet he hunts down his wife’s killer across weeks. How? <strong>Tattoos.</strong> He inks critical facts onto his body — permanent, immutable, always there when he wakes up. Your Temporal Worker is Leonard.</p><p>The Tattoos (Event History)<br>Every time your workflow does something — calls an API, gets a result, receives a signal — Temporal tattoos it into a permanent event log. The Worker can die, restart, or be replaced. The tattoos remain.</p><p><strong>TL;DR</strong><br>Temporal lets you write long-running, failure-prone code as simple async/await functions. Your code can pause for days, survive crashes, and resume exactly where it left off — no state machines, no database columns, no retry logic. This post shows how to build an AI Research Agent that orchestrates LLM calls, handles failures automatically, and waits for human approval without any plumbing code.</p><h3>AI Agents issues in workflow</h3><p>You’ve seen the AI agent demos. An LLM breaks down a task, calls some APIs, synthesizes the results. Looks magical in a 2-minute video. Then you try to ship it to production and realize:</p><p>- The OpenAI API rate-limits you at step 3 of 7<br>- Your server restarts overnight and the agent forgets what it was doing<br>- A user needs to approve something, but you have no way to “pause” for human input<br>- You have no idea which of your 47 running agents is stuck and where</p><p>Welcome to the world of **long-running, failure-prone processes**. State machines were supposed to solve this. They don’t. Let me show you why, and what actually works.</p><h3>The State Machine Trap</h3><p>Here’s how most developers try to build a multi-step AI agent. You create a “state” for every pause point:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_EHgzXAxcCXRbXdOpqjYxg.png" /></figure><h3>Problems:</h3><p>- Every “pause point” needs a database column and explicit state transition<br>- Error handling becomes a nightmare of edge cases<br>- Business logic gets buried under state plumbing<br>- Testing requires mocking the entire state machine</p><p>Now imagine this for an AI agent with 7 steps, branching logic, retry policies, and human approvals. Good luck.</p><h3>The Temporal Approach: Write Code Like Failures Don’t Exist</h3><p>Temporal lets you write the same logic as a simple async function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKJtja60edfSn-I-vXmrmA.png" /></figure><p>This code can:</p><p>- <strong>Pause for days </strong>at the `wait_condition` line<br>- <strong>Automatically retry</strong> on transient failures (rate limits, timeouts)<br>- <strong>Resume from exactly where it left off</strong> if the server crashes<br>- <strong>Scale to millions</strong> of concurrent workflows<br>No state machine. No database columns. No event handlers. Just code.</p><h3>The Replay Process with Memento analogy</h3><p><strong>Scene 1: The Workflow Starts<br></strong>- The code calls `research_topic(topic)`<br>- The Worker executes the LLM call, gets findings<br>- Temporal tattoos: _”Called research_topic → returned [findings]”_<br>- The code hits `await`. The Worker blacks out. Zero memory used.</p><p><strong>(3 Days Pass…)</strong></p><p><strong>Scene 2: Leonard Wakes Up<br></strong>- A human sends an approval signal<br>- Temporal spins up a new Worker (Leonard wakes up in a motel room)<br>- The Worker has no idea what’s happening. It starts the code from line 1.<br>- Line 1 (`research_topic`): Worker checks its tattoos: “Did I do this?”<br>- Tattoo says: “Yes. 3 days ago. Result: [findings].”<br>- Worker: “I won’t call the API again. I’ll just use the tattooed result.” <em>(This is Replay)</em></p><p>- Line 2 (`synthesize_findings`): Checks tattoos → “Already done. Result: draft.”<br>- Line 3 (`wait_condition`): “Approval signal received?” → Tattoo says yes → Continue.</p><p>The Worker replayed its history in milliseconds, reconstructed its state, and picked up exactly where the previous Worker left off. Just like Leonard reading his tattoos to figure out where he is.</p><h3>Why This Is Genius</h3><p>- <strong>Resilience</strong>: Server explodes? Tattoos are on disk. New Worker wakes up, reads them, continues.<br>- <strong>Implicit State</strong>: No database column for `CURRENT_STATE`. State <em>_is_</em> the replay position — “I’ve replayed through tattoo #47, so I’m at step 48.”<br>- <strong>Efficiency</strong>: While waiting for that approval, your code isn’t “paused” — it’s completely dead. You only pay for the milliseconds Leonard is awake and reading his tattoos.</p><h3>Case Study: Why Temporal for AI Agents</h3><p>AI agents have the exact problems Temporal solves:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Man81zGiIP_nBwMJ_ExmA.png" /></figure><p>Let me show you a real example: a Research Agent that breaks down a topic, searches the web, synthesizes a report, and waits for human approval.</p><h3>Building a Research Agent with Temporal</h3><p><strong>The Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6Wp6S7qRt8YtEvEw3d8_g.png" /></figure><p><strong>The Workflow — Signals &amp; Queries:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uscvPX13UKbiTVp46j886g.png" /></figure><p>The Workflow — Main Logic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GU-K-w7XFqW7bPraMWI_w.png" /></figure><h3>Key Temporal Primitives Demonstrated</h3><p><strong>1. Activities</strong> (`@activity.defn`): Functions that can fail. Wrapped with retry policies.<br><strong>2. Workflow</strong> (`@workflow.defn`): The durable orchestration logic. Survives crashes.<br><strong>3. Signals</strong> (`@workflow.signal`): Send events to a running workflow (human approval).<br><strong>4. Queries</strong> (`@workflow.query`): Read workflow state without modifying it.<br><strong>5. wait_condition</strong>: Pause until something happens (or timeout).</p><h3>Activities: The Risky Code</h3><p>Activities are where you put failure-prone code — API calls, LLM requests, database writes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4_ZKFxAZCVGLmu4vXVTSWQ.png" /></figure><p>If OpenAI rate-limits you, Temporal automatically retries with backoff. If your worker crashes mid-request, another worker picks up and retries. You don’t write any of that logic.</p><h3>The Money Shot: Kill the Worker, Workflow Survives</h3><p>Here’s the demo that makes Temporal click:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mLihbkBJMd1WckCxrPTWIQ.png" /></figure><p>The workflow ran for minutes, got killed, resumed, waited for human input, and completed. <strong>Zero state management code written.</strong></p><h3>When NOT to Use Temporal</h3><p>Temporal isn’t for everything. Skip it when:<br>- <strong>Simple request/response</strong>: A REST endpoint that returns in 100ms doesn’t need durable execution<br>- <strong>Pure data pipelines</strong>: Use Spark/Flink for batch analytics<br>- <strong>You need sub-millisecond latency</strong>: Temporal adds overhead for durability</p><h3>Use Temporal when:</h3><p>- Processes span minutes, hours, or days<br>- You need human approval steps<br>- External APIs can fail and you need reliable retries<br>- You need to know “where is this process right now?”<br>- Crashes shouldn’t mean lost work</p><h3>Getting Started</h3><blockquote># Start local Temporal server<br>temporal server start-dev</blockquote><blockquote># Install dependencies<br>pip install temporalio openai httpx</blockquote><blockquote># Run the worker<br>python -m src.worker</blockquote><blockquote># Start a research workflow<br>python -m src.client start “Your topic here”</blockquote><p>The full code for this Research Agent is available at: <a href="https://gist.github.com/libvarun/a226f564f6356606aede862edbac98cd">https://gist.github.com/libvarun/a226f564f6356606aede862edbac98cd</a></p><p>Fork it, break it, rebuild it.</p><h3>Conclusion</h3><p>Temporal fundamentally changes how you think about long-running processes. Instead of:</p><p>- Designing state machines<br>- Building retry logic<br>- Managing “current state” columns<br>- Praying nothing crashes</p><p>You write:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PC5onHfQpnu8OhC80OtrLw.png" /></figure><p>And it just works. Even if the server explodes mid-sleep.</p><p>For AI agents — with their LLM calls, human approvals, and multi-step pipelines — this is a game changer. Stop building the plumbing. Start building the product.</p><p><em>Check out [temporal.io](https://temporal.io) and the [Python SDK docs](https://docs.temporal.io/develop/python) to go deeper.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2767cefac46f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataHub Upgrades 101: Why Your Search “Disappeared” (And How to Fix It)]]></title>
            <link>https://medium.com/@varun.patil578/datahub-upgrades-101-why-your-search-disappeared-and-how-to-fix-it-8391a4004d23?source=rss-1c5d61b74dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/8391a4004d23</guid>
            <category><![CDATA[datahub]]></category>
            <category><![CDATA[metadata-management]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[varun patil]]></dc:creator>
            <pubDate>Sun, 07 Dec 2025 11:23:06 GMT</pubDate>
            <atom:updated>2025-12-07T11:54:29.514Z</atom:updated>
            <content:encoded><![CDATA[<h3>So, you decided to upgrade DataHub. You went from version 0.14 to 1.2.x. You restarted the containers, logged in, and… nothing.</h3><p>Zero search results.<br>The lineage graph is empty.<br>It looks like all your data is gone.</p><p><strong>Don’t panic. Your data is safe.</strong> You just need to re-introduce your database to your search bar.</p><p>This guide is for anyone new to DataHub internals who wants to understand <strong><em>why</em></strong> this happens and exactly how to fix it without losing sleep.</p><h3><strong>The Analogy: The Library and the Card Catalog</strong></h3><p>To understand DataHub, think of a physical library.</p><p>1. The Books (Your Database): This is the actual data (MySQL or PostgreSQL). It holds the “<strong>truth</strong>” — every dataset, user, and tag you’ve ever added. This is permanent.</p><p>2. The Card Catalog (Elasticsearch): This is the search tool. It doesn’t hold the books; it just holds a fast, organized list of <strong><em>where</em></strong> the books are so you can find them quickly.</p><h3>What happened during the upgrade?</h3><p>When you upgraded from v0.14 to v1.2, you essentially hired a new Head Librarian.</p><p>This new Librarian (DataHub v1.2) has a completely new way of organizing the Card Catalog. They don’t understand the old system. When you log in and type a search query, the Librarian looks at the Card Catalog and sees “gibberish” (or nothing at all), so they tell you: “Sorry, no results found.”</p><p>But the <strong>Books</strong> <strong>(your data)</strong> are still sitting safely on the shelves in the back room (the Database).</p><h3>The Solution: RestoreIndices</h3><p>We don’t need to re-create the data. We just need to ask the new Librarian to walk into the back room, read every single book, and write a fresh Card Catalog that they understand.</p><p>In DataHub terms, this process is called <strong>RestoreIndices</strong>.<br>It takes the “Aspects” (data) from the Database and “Re-indexes” them into Elasticsearch.</p><h3>How to Fix It (Step-by-Step)</h3><p>You don’t need to be a coding wizard to run this. It’s a standard operation.</p><p>If you use Docker (Locally or on a VM)<br>We run a special “upgrade job” that does the cleaning for us.</p><p>1. Stop your DataHub containers (just to be safe).<br>2. Run the “clean” command:</p><blockquote>docker-compose -p datahub run — rm datahub-upgrade -u RestoreIndices -a clean</blockquote><p>- <strong>`-u RestoreIndices`</strong>: Tells DataHub “I want to restore my search index.”<br>- <strong>`-a clean`</strong>: Tells DataHub “Throw away the old, broken card catalog first.”</p><h3>If you use Kubernetes or OpenShift</h3><p>If you are running this in a production environment (like RHOS), you can’t just run docker commands. You need to create a <strong>Kubernetes Job</strong>.</p><p>Don’t worry, the DataHub team prepared a template for this. You just need to trigger it.</p><blockquote>kubectl create job — from=cronjob/datahub-cleanup-job-restore-indices restore-indices-manual</blockquote><p><strong>Note</strong>: The exact CronJob name depends on your Helm release name. Run kubectl get cronjobs first to find the one ending in -restore-indices.</p><h3>What to Watch For</h3><p>When you run this command, you will see logs scrolling by. It will look like this:</p><blockquote>… Restoring urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD) …<br>… Successfully restored 500 aspects …</blockquote><p>This is the Librarian walking through the shelves and writing down the new cards. Once it finishes, restart your DataHub services.</p><h3>Conclusion</h3><p>Upgrading complex systems can be scary, especially when screens go blank. But with DataHub, remember: The Database is the Single Source of Truth. As long as your MySQL/Postgres database is healthy, your data is safe. The Search Index is just a view — one that you can rebuild anytime you need to.</p><p>Welcome to DataHub v1.2!</p><h3>Further Reading</h3><p>For advanced configuration options (time-based filtering, batch sizes, parallel processing, Kubernetes Helm values), see the official DataHub documentation: <a href="https://docs-website-t9sv4w3gr-acryldata.vercel.app/docs/how/restore-indices">Restoring Search and Graph Indices from Local Database</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8391a4004d23" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand DataHub with a sample case study]]></title>
            <link>https://medium.com/@varun.patil578/understand-datahub-with-a-sample-case-study-b9ecfcbc2896?source=rss-1c5d61b74dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9ecfcbc2896</guid>
            <category><![CDATA[metadata-management]]></category>
            <category><![CDATA[data-management-platform]]></category>
            <dc:creator><![CDATA[varun patil]]></dc:creator>
            <pubDate>Thu, 27 Nov 2025 03:59:57 GMT</pubDate>
            <atom:updated>2025-11-27T06:09:47.527Z</atom:updated>
            <content:encoded><![CDATA[<h3>Solving Monday Morning Data Crises with Modern Metadata Management</h3><p>It’s Monday morning at ShopSmart(a fictional name), a fast-growing e-commerce company. Three urgent requests hit your desk simultaneously:</p><p><strong>Marketing</strong>: “Which dataset has customer purchase history? We need it for a campaign by end of day.”<br><strong>Engineering</strong>: “A production table changed. What dashboards and ML models will break?”<br><strong>Compliance</strong>: “Where is all our PII data? We need to audit it for GDPR.”</p><p>Without a modern data catalog, these questions would take hours — or even days — to answer. But with DataHub, ShopSmart’s data team can answer them in minutes.</p><p>In this case study, we’ll walk through how DataHub transformed ShopSmart’s data operations, from discovery to governance, using real examples from a live DataHub instance.</p><h3>The Problem: <strong>Data Chaos in a Growing Company.</strong></h3><p><strong>Before DataHub</strong>: <br>ShopSmart had been growing rapidly. Their data ecosystem expanded from a few databases to:<br>- 50+ data sources (PostgreSQL, MySQL, MongoDB, Kafka, S3)<br>- 200+ datasets across multiple data warehouses<br>- 30+ dashboards in Tableau and Looker<br>- 15+ ML models in production<br>- Multiple data pipelines running on Airflow</p><p>The challenges were real:</p><p><strong>1. Data Discovery</strong>: Finding the right dataset meant asking multiple people, searching Slack, or digging through outdated wikis<br><strong>2. Impact Analysis</strong>: Understanding what would break when changing a schema required manual code reviews<br><strong>3. Governance</strong>: Tracking PII data, ownership, and compliance requirements was spreadsheet-based and error-prone<br><strong>4. Documentation</strong>: Knowledge lived in people’s heads or scattered across different tools<br><strong>5. Onboarding</strong>: New team members took weeks to understand the data landscape</p><p>The cost: Data analysts spent 40% of their time just finding and understanding data, not analyzing it.</p><h3>The Solution: DataHub</h3><p><strong>DataHub</strong> is a modern data catalog that provides:<br>- <strong>Unified Search</strong>: Find any data asset across your entire ecosystem<br>- <strong>Data Lineage</strong>: Visualize dependencies and understand impact<br>- <strong>Metadata Management</strong>: Track ownership, documentation, tags, and governance policies<br>- <strong>Self-Service Discovery</strong>: Enable teams to find and understand data independently</p><p>Let’s see how ShopSmart uses DataHub to solve their Monday morning crises.</p><h3>Case Study: Solving ShopSmart’s Data Challenges</h3><h3><strong>Challenge 1: Finding Customer Purchase Data</strong></h3><p>The Request: Marketing needs customer purchase history for a campaign.</p><blockquote>Without DataHub:<br>- Ask 5 different engineers<br>- Search through Slack messages<br>- Check multiple documentation wikis<br>- Time: 2 hours</blockquote><blockquote>With DataHub:<br>- Type “user” in the search bar<br>- Instantly see all related datasets, pipelines, and tasks<br>- Time: 2 minutes</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*McgdCiTelWDLa5vle3Nq3w.png" /></figure><p><em>The search interface shows all assets related to “user” — datasets, pipelines, and tasks.</em></p><p>As you can see in the screenshot above, DataHub’s search instantly surfaces:<br>- Datasets: fct_users_created, fct_users_deleted, logging_event<br>- Pipelines: User Pipeline<br>- Tasks: User Creation Task, User Deletion Task</p><p>Clicking on fct_users_created takes us to the dataset details page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYTxvY_x1iJcSuv0zQ7dIg.png" /></figure><p><em>The dataset details page shows schema, documentation, ownership, and lineage at a glance.</em></p><p>Key Information at a Glance:<br>- Schema: 2 columns (user_id, user_name)<br>- Documentation: “table containing all the users created on a single day”<br>- Owners: John Doe, DataHub (know who to contact)<br>- Lineage: See upstream and downstream dependencies<br>- Statistics: Row counts, size, freshness</p><blockquote>Result: Marketing found the right dataset in 2 minutes instead of 2 hours, and they understood what it contained without needing to ask an engineer.</blockquote><h3><strong>Challenge 2: Impact Analysis for Schema Changes</strong></h3><p>The Request: Engineering wants to change the schema of fct_users_created. What will break?</p><blockquote>Without DataHub:<br>- Manual code review across multiple repositories<br>- Guesswork about downstream dependencies<br>- Risk of breaking production systems<br>- Time: Days of investigation</blockquote><blockquote>With DataHub:<br>- Click the “Lineage” tab<br>- Instantly see all upstream and downstream dependencies<br>- Time: Seconds</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ikrwUcdVpUvUHEXNmTETgg.png" /></figure><p>The lineage graph visualizes data dependencies. The central node is fct_users_created, with upstream sources on the left and downstream consumers on the right.</p><p>What the Lineage View Reveals:</p><blockquote>Upstream (Where data comes from):<br>- logging_events → feeds into → fct_users_created<br>- Shows: “Depends on 1 task, 1 dataset”</blockquote><blockquote>Downstream (What depends on this):<br>- fct_users_created → feeds into → regions, user_name, and 4 other features<br>- Shows: “Used by 4 features, 3 ML primary keys, 1 dataset”</blockquote><p>Real-World Scenario:<br>Engineering: “We need to change the schema of fct_users_created”<br>Data Team: <em>_Opens lineage view_<br></em>Data Team: “Wait! This table feeds into 4 ML features and 3 primary keys. We need to coordinate with the ML team first. Let me check the impact analysis…”</p><blockquote>Result: Engineering prevented breaking production systems by understanding dependencies before making changes. The ML team was notified, and they coordinated the schema change properly.</blockquote><h3><strong>Challenge 3: Compliance and Governance</strong></h3><p>The Request: Compliance needs to audit all PII data for GDPR.</p><blockquote>Without DataHub:<br>- Manual spreadsheet tracking<br>- Outdated information<br>- Missing datasets<br>- Time: Days of work</blockquote><blockquote>With DataHub:<br>- Filter by tag “PII”<br>- See all PII datasets with owners<br>- Export list for audit<br>- Time: Minutes</blockquote><p>Governance Features:<br>1. Tags: Label datasets (e.g., “PII”, “Marketing”, “Production”)<br>2. Domains: Group related assets (e.g., “Customer Data”, “Analytics”)<br>3. Ownership: Assign data stewards who are responsible<br>4. Glossary Terms: Standardize business vocabulary</p><blockquote>Result: Compliance team completed the audit in minutes instead of days, with accurate, up-to-date information.</blockquote><h3>Key Features That Transformed ShopSmart</h3><p><strong>1. Unified Search and Discovery</strong></p><p>DataHub’s search is powerful and intuitive. It searches across:<br>- Datasets: Tables, files, streams<br>- Dashboards: Tableau, Looker, Power BI<br>- ML Models: Model registries<br>- Pipelines: Airflow DAGs, data flows<br>- People: Data owners and stewards</p><blockquote>Impact: Reduced time to find data from hours to minutes.</blockquote><p><strong>2. Visual Data Lineage</strong></p><p>The lineage graph shows:<br>- Upstream: Where data comes from (source systems, transformations)<br>- Downstream: What depends on this data (dashboards, ML models, reports)<br>- Impact Analysis: Understand what breaks when you change something</p><blockquote>Impact: Prevented production incidents by understanding dependencies before changes.</blockquote><p><strong>3. Rich Metadata Management</strong></p><p>Every asset in DataHub has:<br>- Documentation: Descriptions at dataset and column levels<br>- Ownership: Know who to contact<br>- Tags: Categorize and organize<br>- Glossary Terms: Standardize business language<br>- Statistics: Row counts, freshness, size<br>- Schema: Column names, types, descriptions</p><blockquote>Impact: Self-service data understanding without needing to ask engineers.</blockquote><p><strong>4. Governance and Compliance</strong></p><p>DataHub enables:<br>- PII Tracking: Tag and find all PII data<br>- Data Domains: Organize by business function<br>- Ownership: Assign data stewards<br>- Access Policies: Control who can see what</p><blockquote>Impact: Faster compliance audits and better data governance.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*usX9Yjw7NchYhHFI1ENYMg.png" /></figure><h3><strong>Real-World Impact</strong></h3><p><strong>For Data Analysts</strong><br>- Before: Spent 40% of time finding data, 60% analyzing<br>- After: Spent 5% of time finding data, 95% analyzing<br><strong>Result: 8x increase in productivity</strong></p><p><strong>For Engineers<br></strong>- Before: Manual impact analysis, risky schema changes<br>- After: Instant lineage view, safe changes<br><strong>Result: Zero production incidents from schema changes</strong></p><p><strong>For Compliance<br></strong>- Before: Days to complete audits<br>- After: Minutes to complete audits<br><strong>- Result: Faster compliance, reduced risk</strong></p><p><strong>For the Business<br></strong>- Before: Slow decision-making due to data discovery delays<br>- After: Fast decision-making with instant data access<br><strong>Result: Competitive advantage through data-driven decisions</strong></p><h3>Getting Started with DataHub</h3><p># Quick Start<br>DataHub can be up and running in minutes:</p><blockquote># Install DataHub<br>pip install acryl-datahub</blockquote><blockquote># Start DataHub locally<br>datahub docker quickstart</blockquote><blockquote># Ingest sample data<br>datahub docker ingest-sample-data</blockquote><p>Access DataHub at <a href="http://localhost:9002/">http://localhost:9002</a> (default credentials: datahub/datahub).</p><p><strong>Next Steps</strong></p><p>1. Connect Your Data Sources: Use DataHub’s 50+ connectors to ingest metadata from your databases, data warehouses, and tools<br>2. Add Documentation: Start documenting your most important datasets<br>3. Set Up Ownership: Assign owners to critical datasets<br>4. Create Tags: Organize your data with tags (e.g., “PII”, “Marketing”, “Production”)<br>5. Explore Lineage: Understand your data dependencies</p><h3><strong>Why DataHub?</strong></h3><p><strong># Market Leadership</strong><br>- 3,000+ enterprises using DataHub<br>- 13,000+ community members in the open-source project<br>- Trusted by: Apple, CVS Health, Netflix, Visa<br>- Market growth: Context platform market growing from $1B to $9B in 5 years</p><p><strong># Open Source</strong><br>- Free and open source: No vendor lock-in<br>- Active community: 2,300+ pull requests from 200+ contributors<br>- Extensible: Build custom connectors and integrations</p><p><strong># Enterprise Ready<br></strong>- Scalable: Handles millions of metadata entities<br>- Secure: Enterprise-grade security and access controls<br>- Managed option: DataHub Cloud for fully managed deployments</p><h3><strong>Conclusion</strong></h3><p>ShopSmart’s transformation with DataHub demonstrates the power of modern metadata management. By providing:</p><p>- Instant data discovery<br>- Visual lineage and impact analysis<br>- Rich metadata and documentation<br>- Governance and compliance tools</p><p>DataHub enabled ShopSmart to:<br>- Reduce time spent finding data by 8x<br>- Prevent production incidents through better impact analysis<br>- Complete compliance audits 100x faster<br>- Increase data analyst productivity by 8x</p><blockquote>The bottom line: DataHub transforms data from a liability into a competitive advantage.</blockquote><p>Whether you’re a data analyst trying to find the right dataset, an engineer making schema changes, or a compliance officer auditing data, DataHub provides the tools you need to succeed in the modern data landscape.</p><p>Resources<br>- Documentation: <a href="https://docs.datahub.com/">https://docs.datahub.com</a><br>- Community Slack: <a href="https://datahub.com/slack">https://datahub.com/slack</a><br>- GitHub: <a href="https://github.com/datahub-project/datahub">https://github.com/datahub-project/datahub</a><br>- Blog: <a href="https://datahub.com/blog">https://datahub.com/blog</a></p><p>Ready to transform your data operations? Get started with DataHub today at <a href="https://datahub.com/">https://datahub.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9ecfcbc2896" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nvidia’s Greatest Threat Isn’t What You Think: It’s Google’s TPU]]></title>
            <link>https://medium.com/@varun.patil578/nvidias-greatest-threat-isn-t-what-you-think-it-s-google-s-tpu-1a47d0f90926?source=rss-1c5d61b74dbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a47d0f90926</guid>
            <dc:creator><![CDATA[varun patil]]></dc:creator>
            <pubDate>Tue, 12 Aug 2025 07:31:43 GMT</pubDate>
            <atom:updated>2025-08-12T07:31:43.625Z</atom:updated>
            <content:encoded><![CDATA[<p>For years, Nvidia has been the undisputed king of AI hardware. Its powerful GPUs and, more importantly, its <strong>CUDA software ecosystem</strong>, have made it the default choice for virtually every AI researcher and company. This dominance has created a unique dynamic: an <strong>“Nvidia tax”</strong> on the entire industry, where every major player — from startups to cloud giants — must pay a premium for access to this essential technology.</p><p>But this widely accepted view overlooks a powerful, quiet force. While everyone is focused on who can build the next “Nvidia killer” GPU, Google has been playing a different, more strategic game.</p><h3>The TPU: Google’s Vertical Integration Secret Weapon</h3><p>Introduce the <strong>Tensor Processing Unit (TPU)</strong> not just as another chip, but as a core component of Google’s holistic AI strategy.</p><ul><li><strong>What is a TPU?</strong> Briefly explain that a TPU is an <strong>Application-Specific Integrated Circuit (ASIC)</strong>, custom-designed by Google specifically for the massive matrix multiplications at the heart of machine learning. Unlike a general-purpose GPU, a TPU is a highly specialised engine built for a single purpose: accelerating AI workloads.</li><li><strong>The Full-Stack Advantage:</strong> This is the key point. Google doesn’t sell TPUs. Instead, it has created a fully integrated AI stack — co-designing the hardware, software (TensorFlow, JAX), and the vast data center infrastructure all at once. This vertical integration allows Google to extract maximum performance and efficiency from its hardware in a way that simply isn’t possible for a company using off-the-shelf components.</li></ul><p>The AI hardware market is currently a de facto monopoly. NVIDIA’s CUDA software ecosystem and its powerful GPUs have created a deep and wide moat, leaving competitors scrambling. This dominance, while a testament to NVIDIA’s innovation, creates a bottleneck for the entire industry, leading to high prices, supply constraints, and a chilling effect on innovation. The world’s reliance on a single vendor for the most critical component of the AI revolution is a precarious position.</p><h3>Google’s Hidden Weapon: The Trillium TPU</h3><p>Google’s Trillium TPU is not a mere alternative; it’s a powerhouse specifically designed for the demands of large-scale AI. While NVIDIA’s H200 boasts impressive specifications with 141GB of HBM3e memory and significant performance gains over its predecessor, Google’s TPU v6, codenamed Trillium, is engineered for hyper-efficiency in training and inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KiK-a-zJj3insvDiulkIMg.png" /></figure><p>The key advantage of the Trillium TPU lies in its <strong>holistic design</strong>. Google has had the unique advantage of co-designing its hardware, software (TensorFlow, JAX), and AI models. This tight integration allows for unparalleled performance and efficiency on Google’s own workloads. By opening this ecosystem to the world, Google can offer a compelling, and potentially more cost-effective, alternative to NVIDIA’s offerings.</p><h3>Conclusion: A Battle of Ecosystems, Not Chips</h3><p>The AI hardware market is at a crossroads. For years, the conversation has centered on who can build the next “Nvidia killer,” a faster, more powerful GPU to challenge the status quo. Your analysis, however, reveals a far more nuanced and potent threat. The real competition isn’t a battle of chips; it’s a battle of ecosystems.</p><p>Google’s Trillium TPU, and its full-stack approach, represents a fundamental shift in strategy. By tightly integrating hardware and software from the ground up, Google has built an engine of unparalleled efficiency for its own AI needs. This vertical integration is a powerful moat that allows Google to bypass the “Nvidia tax” and potentially offer a more compelling, cost-effective alternative to the rest of the world.</p><p>While Nvidia’s CUDA ecosystem remains a formidable moat, Google’s willingness to open its own optimized platform could be a pivotal moment. The future of AI is not just about raw compute power; it’s about a seamless, integrated, and efficient infrastructure. By challenging the very foundation of how AI is built and deployed, Google’s TPU isn’t just a threat to Nvidia — it’s a potential catalyst for a new era of innovation, where the true winner will be decided not by a single product, but by the strength of its entire ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a47d0f90926" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>