<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hugo Zanini on Medium]]></title>
        <description><![CDATA[Stories by Hugo Zanini on Medium]]></description>
        <link>https://medium.com/@zaninihugo?source=rss-8f4c63386bac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W54mSiInGP5NiZF5PuA3Ig.jpeg</url>
            <title>Stories by Hugo Zanini on Medium</title>
            <link>https://medium.com/@zaninihugo?source=rss-8f4c63386bac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:43:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zaninihugo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nano: an AI grocery assistant, and the lakehouse under it]]></title>
            <link>https://medium.com/data-science-collective/nano-an-ai-grocery-assistant-and-the-lakehouse-under-it-174f6d0343b2?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/174f6d0343b2</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[whatsapp-bot]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 07:27:01 GMT</pubDate>
            <atom:updated>2026-09-30T07:27:01.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmY_bDGFd_G60kyONC-71w.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>My wife and I do one big supermarket run a week. For a long time the list for it lived in our WhatsApp conversation, which is another way of saying it did not really live anywhere.</p><p>Someone would write “we’re out of coffee” on a Tuesday. By Saturday, that message sat twenty messages up, buried between a photo of something one of us had seen and a conversation about the weekend. We would stand in an aisle scrolling backwards through the chat, trying to reconstruct a list out of fragments, and still come home without the coffee 😅 .</p><p>So I built Nano: a small assistant that lives in the same WhatsApp conversation we were already having, keeps one shared list for both of us, and remembers what we bought.</p><p>I also had a second motive: I wanted to learn Google’s open-format lakehouse tooling (Apache Iceberg tables in BigQuery ) and I have never learned anything properly from a tutorial dataset. Our groceries were data I actually cared about being right, at a volume where being wrong cost nothing. That turned out to be the ideal teacher.</p><h3>The problem, and the one that came after it</h3><p>The first problem was simply that a chat thread is not a list. It has no state. Nothing in it can be ticked off, and by the time you need it, it has scrolled away.</p><p>The obvious answer is to put a bot in the group chat, and that answer is not available. WhatsApp’s Business Platform does not let you add an API number to an existing personal group. So Nano runs as <strong>two separate one-to-one chats sharing a single backend</strong>. When either of us changes the list, both phones hear about it - it behaves like a shared assistant.</p><p>The second problem arrived with the first receipt, and it is the one that shaped a few decisions in the product.</p><p>We are living in Barcelona, and our receipts print in Catalan. A line reads MADUIXA. Another reads PASTANAGA. A third says PIT FAM., which even I had to think about. The questions I wanted to ask were in English — <em>how much do we spend on strawberries?</em> — and no amount of clever database searching will ever connect &quot;strawberries&quot; to MADUIXA. The two words do not share a single letter.</p><p>That led to an important design decision: whatever I built had to understand each line <strong>at the moment it was stored</strong>, not at the moment it was asked about. Translation is part of writing the data.</p><h3>What it actually does</h3><p>Everything happens in WhatsApp in natural language. There are no commands to memorise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43dbef5607a9357e03dcdf153636f6e3/href">https://medium.com/media/43dbef5607a9357e03dcdf153636f6e3/href</a></iframe><p>Here are a few examples of Nano being used in the day to day:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*vDVfRHnzpPyz3GxME7rx0Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*U2nWvAgr675WgwkUPiBv4w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*OHlyVzWr1da5Zw7EDM_84g.jpeg" /><figcaption>Nano in action: Image by the author</figcaption></figure><h3>High level architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnS9b9T9bEKrO-b2JVWHSg.png" /><figcaption>Nano high-level architture: Image by the author</figcaption></figure><p>The webhook has exactly one job: decide in milliseconds whether an incoming message is genuine and worth processing, hand it to <a href="https://cloud.google.com/tasks/docs">Cloud Tasks</a>, and return 200. Everything slow happens somewhere else.</p><p>Reading the messages themselves uses <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling">Gemini’s function calling</a>. The model does not write the reply; it chooses one of <a href="https://github.com/hugozanini/nano/blob/main/app/nano/gemini.py#L117-L390">fourteen pre-defined functions</a> and fills in its arguments, and the code decides what to do. Receipt photos go to the same model, so there is no separate OCR service anywhere in the stack.</p><h3>Why a lakehouse for Nano</h3><p>There are two kinds of data here, and they want completely different things.</p><p>The <strong>live shopping list</strong> is read and written constantly, is a few dozen words long, and has no analytical value whatsoever on its own. It needs to be fast and simple, and it lives in <a href="https://cloud.google.com/firestore/docs">Firestore</a>.</p><p>The <strong>purchase history</strong> is the opposite. It is written once per shopping trip, never modified, and exists entirely so that questions can be asked of it later. That is the part worth putting in a lakehouse: six <a href="https://cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery">Apache Iceberg managed tables</a> in BigQuery, which until recently were called “BigLake tables for Apache Iceberg”.</p><p>If Iceberg is unfamiliar for you, the short version is this:</p><blockquote>It is an open table format: your data sits as ordinary Parquet files in a bucket <strong>you own</strong>, alongside metadata files describing the shape of the table. BigQuery gives you the fully managed experience — plain SQL, no servers — but the files are yours, in your bucket, in a format other engines can read.</blockquote><p>Creating one is table is quite easy, here is an example:</p><pre>CREATE TABLE `&lt;dataset&gt;.purchases` (<br>  purchase_id  STRING,<br>  purchased_at TIMESTAMP,<br>  store_name   STRING,<br>  total_amount NUMERIC,<br>  currency     STRING<br>)<br>WITH CONNECTION `&lt;region&gt;.&lt;connection&gt;`<br>OPTIONS (<br>  table_format = &#39;ICEBERG&#39;,<br>  file_format  = &#39;PARQUET&#39;,<br>  storage_uri  = &#39;gs://&lt;your-bucket&gt;/warehouse/purchases&#39;<br>);</pre><p>You write to it with normal INSERT and MERGE. There is no catalog service to operate. The only piece of plumbing is a <a href="https://cloud.google.com/bigquery/docs/connections-api-intro">BigQuery connection</a> whose service account is allowed to write into your bucket.</p><h4>How the data is modelled</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9yYmQbtmSO6ekHY0B44daQ.png" /><figcaption>Data Model — click for a better view (Image by the author)</figcaption></figure><p>The three main design decisions behind the modeling:</p><p><strong>Every line item keeps three names.</strong> item_name is what the shop printed, item_name_en is the English translation, and normalized_name is the canonical term used for matching. Keeping all three is what lets Nano answer a question about strawberries, reply in English, and still show you the Catalan text that is actually on the receipt.</p><p><strong>stores is a separate table</strong>, and shops are resolved to it once. A supermarket brand prints its name slightly differently across receipts, and without a stable store_id every one of those spellings becomes its own shop, quietly splitting your spending across four rows that should be one.</p><p><strong>Money is stored twice.</strong> total_amount with its currency is what was actually charged. amount_eur is the same figure converted at fx_rate, using the rate from fx_rate_date — the day of the purchase. It means totals from last year still reconcile.</p><p>list_events, question_log and feature_requests have no relationships at all. They are append-only logs, and the last two are what section 6 is about.</p><h3>The loop: Nano files its own backlog</h3><p>Every question made by the user gets logged along with how it was resolved: answered from a hand-written SQL template, answered by SQL the model generated on the spot, or not answered at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgtzSyxbsy1o19tiuWWLLw.png" /><figcaption>Nano self improvement loop: Image by the author</figcaption></figure><p>A question that nothing could answer is a product evolution signal/ feature request: someone wanted something and did not get it. The catch is that nobody files a ticket asking the feature — so Nano files it instead. Requests are deduplicated on meaning, so three different phrasings of the same idea become one row with a count of three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JspAYWuXrIt2Oe8ii6r9aw.png" /><figcaption>Example of a Nano feature request (Image by the author)</figcaption></figure><h3>Try it!</h3><p>Nano is open source at <a href="https://github.com/hugozanini/nano"><strong>github.com/hugozanini/nano</strong></a> — the full application, the Iceberg schema as numbered migrations, and the test suite.</p><p><strong>Contributions are very welcome.</strong> If you want somewhere to start: the Spark job for shopping patterns is unwritten, the analytics layer could cache repeated questions instead of regenerating their SQL, and receipt totals are not yet reconciled against their line items.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=174f6d0343b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/nano-an-ai-grocery-assistant-and-the-lakehouse-under-it-174f6d0343b2">Nano: an AI grocery assistant, and the lakehouse under it</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Refactoring a Web App with Antigravity’s Sandbox Mode and /goals]]></title>
            <link>https://medium.com/data-science-collective/refactoring-a-web-app-with-antigravitys-sandbox-mode-and-goals-dc58ddda94db?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc58ddda94db</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[ai-code-assistant]]></category>
            <category><![CDATA[google-antigravity]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[antigravity-cli]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Sun, 21 Jun 2026 15:58:25 GMT</pubDate>
            <atom:updated>2026-06-24T21:03:54.618Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iNIipb_xFiA5LzUDYRkxFQ.png" /></figure><p>Back in March 2023, I built a little project I’m still proud of: a browser app that runs YOLOv7 for real-time, offline object detection. The algorithms and image processing came out great. The UI… did not.</p><p>It still surprises me how many teams reach for expensive multimodal models to detect objects, one of the most common tasks in the industry, when a model like the YOLOv7 in this repo gives excellent results and runs in real time in almost any phone browser. So I decided to give the repo the refresh it deserves: review my old implementation, bump the libraries, and finally make the interface look good. And I wanted it done in one shot, with an agent running on its own.</p><p>There’s just one thing that always makes me nervous about letting agents run autonomously: what they can do to my machine. I once left agents running for 15 hours and they managed to lock up my own file system. So before going full <em>yolo mode</em>, I needed a safety net.</p><p>That net is Antigravity’s sandbox.</p><h4>Setting up the sandbox</h4><p>If you’re still on Claude Code or Cursor and haven’t tried the new Antigravity CLI yet please do it! Install it:</p><pre>curl -fsSL https://antigravity.google/cli/install.sh | bash</pre><p>Then open ~/.gemini/antigravity-cli/settings.json and turn the sandbox on:</p><pre>{<br>  &quot;enableTerminalSandbox&quot;: true<br>}</pre><h3>The refactoring pront for a 30 min run and zero babysitting</h3><p>With the sandbox active, I handed the agent a single /goal prompt and let it run for about 30 minutes on its own:</p><pre>Refactor and optimize the YOLOv7 TensorFlow.js browser application. <br>Focus on maximizing execution efficiency, cleaning up legacy patterns, <br>upgrading dependencies safely, and building a premium, modern user interface.<br>Modernize and bump libs to safer, modern versions.<br>Optimize inference :improve tensor operations and inference time.<br>Redesign the UI: dark mode, dynamic confidence-threshold sliders, p<br>lay/pause controls, and a real-time performance dashboard (FPS, inference time,<br>WebGL memory) - without sacrificing speed.<br>Make sure the app builds cleanly with npm run build.<br>Open as many branches and PRs as needed, and finish by telling me which <br>branch to test.</pre><p>It only stopped once — to ask whether it could run an npx command that needed network access. That pause was exactly the guardrail I was hoping for when I turned the sandbox on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SoUJD4c4-Ky2-CV4QiVlNw.png" /></figure><h4>Making it elegant, not just “AI-generated”</h4><p>The result was already a big jump from my original code, it also cleared the 23 vulnerabilities the old app was carrying. But it had <em>that classical AI look</em>: the generic, slightly soulless interface you see in every gen ai demo. I wanted something with more taste.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6WgxX_Wv0CnN5OV8FH85w.png" /></figure><p>So I gave Antigravity a second prompt, this time asking it to spin up multiple agents that would review each other’s work until the design felt right:</p><pre>While the application is functional, the current interface is underwhelming <br>and resembles a basic web framework boilerplate. To address this, I want you <br>to spin multiple agents to develop a more sophisticated visualization. The <br>new design should be elegant and align with the style of Gemini and <br>Google design systems. I want a dedicated design agent to create a <br>comprehensive plan, followed by a frontend development agent to handle the <br>implementation within the same branch.</pre><p>Now we’re talking!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bokc7E3XL_ZSG6GkO-LlwQ.png" /></figure><p>One small note: <em>in the demo I dimmed my phone on purpose, to show the model holds up in tough lighting. That robustness is a big reason this repo got popular in industrial setups.</em></p><p>And if you want to go end-to-end and actually interpret the Yolo detections, a Gemma model plugs in nicely on top of this. I&#39;ve seen many cases succeed with this pipeline.</p><p><a href="https://github.com/hugozanini/yolov7-tfjs">Here’s the repo if you want to check the results yourself</a>.</p><p>See you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc58ddda94db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/refactoring-a-web-app-with-antigravitys-sandbox-mode-and-goals-dc58ddda94db">Refactoring a Web App with Antigravity’s Sandbox Mode and /goals</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving Beyond Screen Scraping: Creating an Agent-Native Web App with WebMCP]]></title>
            <link>https://medium.com/data-science-collective/moving-beyond-screen-scraping-creating-an-agent-native-web-app-with-webmcp-4818552e1e11?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/4818552e1e11</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[web-mcp]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Mon, 16 Feb 2026 17:55:07 GMT</pubDate>
            <atom:updated>2026-02-16T17:55:07.212Z</atom:updated>
            <content:encoded><![CDATA[<p>The relationship between humans and the web is evolving. For thirty years, we built the web for eyes and index fingers. We optimized for visual hierarchy, white space, and click-through rates. But a new user has entered the browser: the <strong>AI Agent</strong>.</p><p>Until now, agents have been forced to navigate the web like digital ghosts- using computer vision or fragile DOM scraping to guess what a button does. This is slow, expensive, and prone to hallucinations — until now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JljEP2BiTnySG9445Ng1Yg.png" /><figcaption>Web MCP in action (image by Author)</figcaption></figure><p><strong>Web MCP (Model Context Protocol), </strong>co-developed by engineers at Google and Microsoft, and is now<a href="https://developer.chrome.com/blog/webmcp-epp"> available on early preview</a> under Chromium 146.0.7672.0 or higher, and is a revolutionary proposal that allows us to build pages that are <strong>Agent-Native </strong>and change how web sites interact with the world.</p><p>In this article, I’ll deep-dive into how I transformed a booking website inspired by Airbnb into an agent-ready platform, showing the code, the architecture, and the performance data to prove the benefits of using Web MCP.</p><h4><strong>The Problem: The “Visual Tax” of Current Web Agents</strong></h4><p>When an agent helps you find something on the web today, it often has to process the entire DOM tree of a page or read a screenshot. This “Visual Tax” consists of thousands of tokens of HTML, CSS, and metadata that the agent doesn’t actually need.</p><p><strong>WebMCP</strong> replaces this with a formal contract.</p><p>By exposing structured functions directly to the browser’s model context, we allow the agent to bypass the UI entirely and interact with the application’s core logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44hEuuECt8casPWGBBULng.png" /><figcaption>Agentic Scraping vs WebMCP (image by author)</figcaption></figure><h4><strong>Case Study: “AirBird” (The booking marketplace)</strong></h4><p>With the help of <a href="https://www.magicpatterns.com/">Magic Patterns</a>, I created a booking marketplace called “AirBird,” a React-based web app, and integrated the WebMCP <strong>Imperative API</strong>. Unlike the Declarative API (which uses HTML attributes for simple forms), the Imperative API allows for deep integration with application state —<a href="https://medium.com/medium.com/r?url=https%3A%2F%2Fdeveloper.chrome.com%2Fblog%2Fwebmcp-epp%23structured_interactions_for_the_agentic_web"> m</a><a href="https://developer.chrome.com/blog/webmcp-epp#structured_interactions_for_the_agentic_web">ore about the interaction types here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UquBl--iK9LQ1pvGvwFWBA.png" /><figcaption>Airbird home page (image by author)</figcaption></figure><h3><strong>The Full Agentic Toolbelt</strong></h3><p>To make AirBird fully navigable by an agent, I implemented six atomic tools:</p><p>1. <strong>search_listings</strong>: The entry point. Sets location, dates, and guest count.</p><p>2. <strong>get_available_listings</strong>: The discovery tool. Returns a structured JSON list of visible properties.</p><p>3. <strong>filter_listings</strong>: The precision tool. Refines results by price, amenities, and room counts.</p><p>4. <strong>select_listing</strong>: The navigator. Moves the app state to a specific property’s details page.</p><p>5. <strong>reserve_listing</strong>: The actuable. Completes the booking and triggers the confirmation flow.</p><p>6. <strong>go_back_home</strong>: The reset. Returns the agent to the main search grid.</p><p>Here are two examples of the tools implemented.</p><h4><strong>The Discovery Pattern: get_available_listings</strong></h4><p>The biggest hurdle for web agents is <strong>seeing results</strong>. If the agent calls a search tool, how does it know what was found? I implemented a <em>Discovery Tool</em> that returns a structured JSON payload of currently visible listings.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f017c2a3f0ec5598d0585fb24d70fd8/href">https://medium.com/media/2f017c2a3f0ec5598d0585fb24d70fd8/href</a></iframe><h4><strong>High-Fidelity Actuation: reserve_listing</strong></h4><p>Instead of the agent clicking a <strong>Reserve</strong> button and hoping it works, I provided an explicit booking tool. This tool handles the business logic (calculating fees, checking dates) and returns a structured confirmation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38c64c7b3b32b0b6cf986177b5633058/href">https://medium.com/media/38c64c7b3b32b0b6cf986177b5633058/href</a></iframe><h3><strong>Performance Analysis</strong></h3><p>I ran a full booking flow using the MCP tool inspector using the following prompt:</p><blockquote>Find me an entire home in San Francisco for 3 guests with at least 2 bedrooms and a maximum price of $300 per night, and then reserve it between March 1 and 7</blockquote><p>The video below is the tool in action using gemini-2.5-flash:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*l2_bd1oJWY3jVpOrRdLgVg.gif" /><figcaption>WebMCP in Action (gif from author)</figcaption></figure><p>By analyzing the execution trace, we can see the efficiency of the WebMCP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9GFm8Dl7ReUWCP4IEinJ3g.png" /><figcaption>Image by author</figcaption></figure><p><strong>Tokens breakdown:</strong></p><ul><li><strong>Structured Intent (Tools)</strong>: ~900 tokens avg per call.</li><li><strong>Discovery Content</strong>: ~950 tokens for 37 listings (JSON).</li><li><strong>Total Usage</strong>: <strong>~15,000 tokens</strong> ($0.0045) for a complex, 5-step transaction.</li></ul><p><strong>The Result</strong>: A traditional DOM-scraping agent would likely consume <strong>5x to 10x more tokens</strong> just to re-parse the complex React headers, footers, and recursive component trees at every step. By using WebMCP, we didn’t just make it faster; we made it <strong>more deterministic and cost-efficient</strong>.</p><h4><strong>Dealing with synchronous state</strong></h4><p>During thedevelopment, I discovered a pattern called <strong>State-to-Tool Sync</strong>.</p><p>In React, WebMCP tools must be re-registered within a useEffect that listens to state changes.</p><pre>useEffect(() =&gt; {<br>  // Register tools...<br>}, [selectedListing, filteredListings, searchFilters, filters, activeCategory]);</pre><p>If you don’t include your search results or filters in the dependency array, your AI agent will suffer from <em>stale closures</em>, calling tools that reference an old version of your app state.</p><blockquote><strong><em>Best Practice</em></strong><em>: Your tools are only as good as the state they close over. Always sync your registry with your UI’s </em>Source of Truth<em>.</em></blockquote><h3>How to Build Yourself</h3><p>WebMCP is currently a developer preview. To replicate this on your own machine:</p><ol><li>Use <strong>Chrome 146+</strong> and enable the #enable-webmcp-testing flag.</li><li>Install the <a href="https://chromewebstore.google.com/detail/model-context-tool-inspec/gbpdfapgefenggkahomfgkhfehlcenpd">Model Context Tool Inspector Extension</a>.</li><li>Follow the official <a href="https://github.com/webmachinelearning/webmcp">WebMCP documentation</a>.</li></ol><p>The code used for building Air Bird can be accessed at <a href="https://github.com/hugozanini/air-bird-booking-web-mcp">https://github.com/hugozanini/air-bird-booking-web-mcp</a></p><p>I hope this project inspires you to think about how to use Web MCP to power your next Web App.</p><p>Feel free to reach out on <a href="https://www.linkedin.com/in/hugozanini/">LinkedIn</a> if you have questions, ideas, or just want to talk about #WebAI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4818552e1e11" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/moving-beyond-screen-scraping-creating-an-agent-native-web-app-with-webmcp-4818552e1e11">Moving Beyond Screen Scraping: Creating an Agent-Native Web App with WebMCP</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI-Powered web APP to manage 2K+ Jira support tickets with Gemini and Antigravity]]></title>
            <link>https://medium.com/data-science-collective/building-an-ai-powered-web-app-to-manage-2k-jira-support-tickets-with-gemini-and-antigravity-327e679772a3?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/327e679772a3</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[google-antigravity]]></category>
            <category><![CDATA[jira]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Sun, 21 Dec 2025 23:26:18 GMT</pubDate>
            <atom:updated>2025-12-22T20:03:16.291Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQ-b_LoXV5mrTv7XK2xd3A.png" /><figcaption>Image by Google Nano Banana — AI generated</figcaption></figure><p>As a product manager for an internal data platform serving over 2,000 monthly active users, I face a challenge that many PMs in similar positions know all too well: <strong>drowning in support tickets </strong>🫠.</p><p>My users range from business analysts to software engineers, each with different needs, technical backgrounds, and ways of expressing problems. This diversity creates a wide spectrum of support requests, from “<em>How do I access this dataset</em>?” to “<em>The entire pipeline is down</em>.”</p><p>However, I&#39;m constantly reviewing these tickets. <strong>Analyzing support requests is one of the best ways to identify areas where your product needs improvement</strong>. User pain points surface clearly in support requests. The problem? When you’re managing multiple products, careful analysis becomes very time-consuming.</p><p>My manual process looked like this:</p><ol><li>Export tickets from Jira (clunky, slow interface)</li><li>Copy data into a spreadsheet</li><li>Run Gemini on the CSV to help categorize</li><li>Manually analyze patterns</li><li>Repeat for each support queue</li></ol><p><strong>Total time: ~2.5 hours per queue, every week.</strong></p><blockquote>That’s time I wasn’t spending on product strategy, roadmap planning, or working with my teams on actually solving these problems. I knew there had to be a better way.</blockquote><h4>The Solution: Help Insights</h4><p>I decided to build a system that would automate the entire support ticket analysis workflow for me using <strong>Gemini 3</strong>. I call it <strong>Help Insights</strong> (or <strong>Hi 👋</strong>).</p><p>The core idea is simple:</p><ol><li>Provide a Jira Query (JQL) once for each queue you want to monitor</li><li>Help Insights 👋 automatically fetches and analyzes tickets</li><li>Gemini helps categorize tickets based on patterns</li><li>Get instant insights through an interactive dashboard</li><li>Ask Gemini questions about your tickets in natural language</li></ol><p><strong>Time saved: From 2.5 hours to 10 minutes per analysis session.</strong></p><p>Now, every Friday, I spend one focused hour reviewing all my products, understanding support patterns, briefing on-call engineers, and prioritizing long-term fixes to prevent recurring issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*3I5sRBNDeQkwRRWvDVW6uA.gif" /><figcaption>Image by Author</figcaption></figure><h4>Why This Matters</h4><p>Before Help Insights, I was stuck in a reactive cycle:</p><ul><li>Tickets came in faster than I could analyze them</li><li>Patterns were buried in noise</li><li>Context switching between products was exhausting</li></ul><p>After building Help Insights:</p><ul><li><strong>Proactive pattern detection</strong>: Gemini identifies recurring issues automatically</li><li><strong>One dashboard for everything</strong>: All support queues in one place</li><li><strong>AI-powered categorization</strong>: 2K+ tickets categorized in under a minute</li><li><strong>Data-driven prioritization</strong>: Know exactly what needs fixing</li><li><strong>Time for strategy</strong>: I could focus more on product improvements</li></ul><p>The best part? Every insight is backed by actual user data, making it easy to justify product investments.</p><h4>How It Works</h4><p>Help Insights is a full-stack application designed to run locally (keeping your sensitive data secure):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BfI4zqwY37ySOs9nlt_nA.png" /><figcaption>Image by Author</figcaption></figure><p><strong>Tech Stack:</strong></p><ul><li><strong>Frontend</strong>: React + TypeScript with shadcn/ui (built using Google’s Antigravity)</li><li><strong>Backend</strong>: FastAPI (Python) with SQLite</li><li><strong>AI</strong>: LiteLLM gateway → Gemini models</li><li><strong>Data Source</strong>: Jira REST API</li></ul><p><strong>Why LiteLLM?</strong> Most companies have their own AI gateways for control, monitoring, and cost management. LiteLLM provides a unified interface that works with any company’s setup and is connected with Gemini models (my favorites 🤓).</p><h3>Setting Up Help Insights 👋</h3><p>The entire system runs locally on your machine, ensuring your company’s support data never leaves your control.</p><p><strong>Step 1: Clone and Install</strong></p><pre>git clone https://github.com/hugozanini/help-insights</pre><p><strong>Backend Setup:</strong></p><pre>cd backend<br>python -m venv venv<br>source venv/bin/activate  # Windows: venv\Scripts\activate<br>pip install -r requirements.txt<br>uvicorn main:app --reload --port 8000</pre><p><strong>Frontend Setup:</strong></p><pre>cd frontend<br>npm install<br>npm run dev</pre><p>Open <a href="http://localhost:5173/">http://localhost:5173</a> and you’re ready to configure.</p><p>[IMAGE: Setup wizard welcome screen]</p><h3>Step 2: Configure Jira Connection</h3><p>The first-time setup wizard guides you through configuration.</p><p><strong>Getting Your Jira API Token:</strong></p><ol><li>Go to <a href="https://id.atlassian.com/manage-profile/security/api-tokens">https://id.atlassian.com/manage-profile/security/api-tokens</a></li><li>Click “Create API token”</li><li>Give it a descriptive name</li><li>Copy the token</li></ol><p><strong>What you’ll need:</strong></p><ul><li>Jira instance URL (e.g., yourcompany.atlassian.net)</li><li>Your Atlassian email</li><li>API token</li></ul><p>[IMAGE: Jira configuration step]</p><h3>Step 3: Configure AI (Gemini via LiteLLM)</h3><p>For this tutorial, I’m using Gemini 3 through LiteLLM. Most companies have their own LiteLLM proxies or similar AI gateways for governance and cost control.</p><p><strong>If you have a company LiteLLM proxy</strong> (recommended):</p><ul><li>Ask your platform team for the endpoint URL</li><li>Get your API key</li><li>Use the model name as configured in your proxy</li></ul><p>[IMAGE: LiteLLM configuration step]</p><p><strong>If you’re using Google AI Studio directly:</strong></p><ol><li>Go to <a href="https://aistudio.google.com/">https://aistudio.google.com/</a></li><li>Click “Get API Key”</li><li>Create a new API key in your Google Cloud project</li><li>Use endpoint: <a href="https://generativelanguage.googleapis.com/v1beta">https://generativelanguage.googleapis.com/v1beta</a></li></ol><h3>Step 4: Add Your First Product</h3><p>Now comes the interesting part. Instead of rigid project configurations, you define:</p><ul><li><strong>Product Name</strong>: What you call this queue (e.g., “DataHub Support”)</li><li><strong>Team</strong>: Who manages it (e.g., “Data Governance Team”)</li><li><strong>JQL Query</strong>: How to fetch tickets</li></ul><p><strong>Example queries (JQL):</strong></p><pre>project = &quot;DATAHUB&quot; AND labels = &quot;support&quot;<br>project = &quot;AIRFLOW&quot; AND component = &quot;Platform&quot;<br>project = &quot;SNOWFLAKE&quot; AND &quot;Request Type&quot; = &quot;Help&quot;</pre><p><strong>Load From Date</strong>: How far back to fetch tickets (default: 6 months)</p><p>Click “Preview Tickets” to verify your query works, then “Finish Setup.”</p><p>[IMAGE: Project configuration with JQL]</p><h3>The AI Categorization Magic</h3><p>This is where Gemini shines. Once your tickets are loaded, click “Configure” in the Categories section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gbMlrIMNjZwPMIBigBczA.png" /><figcaption>Image by Author</figcaption></figure><p>Click “Suggest Categories” and watch Gemini analyze your tickets.</p><p>Edit as needed, then click “Preview Categorization.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iq4UFEoX1lFRbYlV55GbyA.png" /><figcaption>Image by Author</figcaption></figure><p>If you are satisfied with the categorization, click “Apply to All Tickets” and wait a few seconds to have all your tickets categorized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pb6jZdyEoVUsCArLuuiLRQ.png" /><figcaption>Image by Author</figcaption></figure><h3>Analyzing Your Support</h3><p>Now the real insights begin. Here is the dashboards with all tickets categorized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ali3gwYMgrFt3Ti6Xo8jBw.png" /><figcaption>Image by Author</figcaption></figure><h4>Talking with Gemini</h4><p><strong>Instead of writing complex queries, just ask Gemini:</strong></p><ul><li>“What are the top issues this week?”</li><li>“Why are tickets taking longer to resolve?”</li><li>“Which category needs immediate attention?”</li></ul><p>Gemini analyzes the filtered tickets and provides context-aware answers in markdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-etaXjSoNee4u29i-B-JQ.png" /><figcaption>Image by Author</figcaption></figure><p>When analyzing the tickets, I also like to add notes with a few comments to guarantee I have already reviewed the request and registered my insights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YHha_yOWlcr75i_neBobHA.png" /><figcaption>Image by Author</figcaption></figure><h3>What&#39;s next</h3><p>Help Insights 👋 is open source and actively evolving. Future enhancements I’m considering, and I&#39;d love to receive PRs implementing them:</p><ul><li><strong>Automated reporting</strong>: Automatic weekly ticket digests</li><li><strong>Slack integration</strong>: Push insights to channels</li><li><strong>Trend predictions</strong>: ML-based forecasting tickets by categories</li></ul><p>But the core value is already there: <strong>turning support ticket chaos into actionable product insights</strong>.</p><h3>Try Help Insights 👋</h3><p>I hope this project inspires you to think about how AI can augment your work as a product manager.</p><p>Feel free to reach out on <a href="https://www.linkedin.com/in/hugozanini/">LinkedIn</a> if you have questions, ideas, or just want to discuss PM tools and AI.</p><h4>Acknowledgments</h4><p>This project was developed as part of the Gemini Sprint Program organized by Google’s Developer Expert community. Special thanks to Google for providing free access to the Gemini API and supporting the open-source AI community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=327e679772a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/building-an-ai-powered-web-app-to-manage-2k-jira-support-tickets-with-gemini-and-antigravity-327e679772a3">Building an AI-Powered web APP to manage 2K+ Jira support tickets with Gemini and Antigravity</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Gemini-powered data catalog extension for Cursor]]></title>
            <link>https://medium.com/data-science-collective/building-a-gemini-powered-data-catalog-extension-for-cursor-539c504a4c23?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/539c504a4c23</guid>
            <category><![CDATA[open-metadata]]></category>
            <category><![CDATA[vscode-extension]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cursor]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 15:15:38 GMT</pubDate>
            <atom:updated>2025-08-10T15:15:38.497Z</atom:updated>
            <content:encoded><![CDATA[<p>Developers spend a lot of time context-switching between their IDE, data catalogs, docs, and metadata tools.</p><p>In this article, I’ll show how I built a Cursor extension that brings data discovery and lineage into the IDE, with Gemini providing natural language search and insights.</p><p>The setup uses OpenMetadata and Airflow locally and a React-based webview inside a Cursor extension.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fkua_mE0_ZKk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dkua_mE0_ZKk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fkua_mE0_ZKk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e276ef3a4b9dfafc053bfcf2dc2ca8d6/href">https://medium.com/media/e276ef3a4b9dfafc053bfcf2dc2ca8d6/href</a></iframe><h4>Why build this</h4><p>I lead Development Experience initiatives in a Data Platform used by thousands of developers. When I ask them what hurts productivity most, the top answer is <strong>context switching</strong>. A typical workflow jumps across tools: data catalog, notebook/SQL, data transformation code, CI, staging runs, monitoring, and documentation — over and over.</p><p>The goal of this project is simple: <strong>do more of this inside the IDE</strong>. Cursor is my daily IDE and a get increasily popular, so I built a Cursor extension that puts data search and lineage next to the terminal, with an experience that feels native to data developers.</p><h4>What we’re building</h4><p>A Cursor extension that:</p><ul><li>Shows a search UI for your data catalog</li><li>Uses Gemini to interpret natural language queries and summarize results</li><li>Displays dataset cards with key metadata</li><li>Renders interactive lineage graphs</li></ul><p>External components</p><ul><li>Local OpenMetadata deployment: catalog, search, and ingestion</li><li>AI integration: Gemini API for NL search and insights</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDyay8Id2qEAacmBewuytQ.png" /><figcaption>Image by author</figcaption></figure><h4>Why the UI lives next to the terminal</h4><p>The terminal is high-traffic and always visible component in an IDE. Adding the extension as a tab next there feels natural and keeps context switching low. Cursor webviews can render full web apps, which gives developers a lot of freedom while staying inside the IDE.</p><h3>Environment setup</h3><p>We’ll run OpenMetadata locally using Docker. This provides:</p><ul><li>MySQL: stores metadata and configuration</li><li>Elasticsearch: powers metadata search</li><li>OpenMetadata Server: API and UI</li><li>Airflow: orchestrates ingestion and quality workflows</li></ul><p>The demo data includes realistic fake datasets such as:</p><ul><li>Customer information</li><li>Orders and transactions</li><li>Product catalog</li><li>User activity logs</li><li>Sales and revenue metrics</li></ul><h4>Setup Process</h4><pre># 1. Create directory<br>mkdir openmetadata-docker<br>cd openmetadata-docker</pre><pre># 2. Download docker-compose file<br>curl -L https://github.com/open-metadata/OpenMetadata/releases/latest/download/docker-compose.yml -o docker-compose.yml<br># 3. Start services<br>docker compose up -d<br># 4. Access OpenMetadata at http://localhost:8585<br># Default credentials: admin@open-metadata.org / admin</pre><p>If everything goes right, you should see this interface for OpenMetadata:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfudpb7lG8JKJkU119ZvOQ.png" /><figcaption>Image by author</figcaption></figure><p>And this for airflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgcqCvqIGGNOJsZy_xijRg.png" /><figcaption>Image by author</figcaption></figure><h4>Cursor Extension Development</h4><p>Cursor is a VS Code extension so all the development here will follow the <a href="https://code.visualstudio.com/api/extension-guides/overview">official VS Code Extension API documentation</a>. As mentioned before, the extensions can interact with basically the whole IDE and are web interfaces. Below is the project structure:</p><pre>open-metadata-cursor-extension/<br>├── src/<br>│   ├── extension.ts                     # Extension entry point<br>│   ├── OpenMetadataExplorerProvider.ts  # Webview provider &amp; orchestration<br>│   └── services/<br>│       ├── OpenMetadataService.ts       # API client with NLP capabilities<br>│       └── GeminiService.ts             # AI integration service<br>├── src/webview/                         # React UI components<br>│   ├── App.tsx                          # Main application component<br>│   ├── components/<br>│   │   ├── SearchInterface.tsx          # Search input with NL examples<br>│   │   ├── AIInsights.tsx               # Streaming AI overview<br>│   │   ├── ResultsList.tsx              # Table results display<br>│   │   ├── TableCard.tsx                # Individual table cards<br>│   │   └── ConfigStatus.tsx             # Configuration indicator<br>│   └── styles.css                       # VS Code-themed styling<br>├── package.json                         # Extension manifest &amp; dependencies<br>├── webpack.config.js                    # Build configuration<br>└── .vscode/                             # VS Code debug configuration<br>    ├── launch.json<br>    └── tasks.json</pre><p>The extension implements four main features. Each component is designed to feel native to the Cursor environment while providing powerful data catalog capabilities.</p><p><strong>Search with Gemini</strong></p><p>The search serves as the primary entry point for the extension, combining traditional keyword search with natural language processing and Gemini-powered analysis. Users can type conversational queries like “<em>What customer information do I have?</em>” or simple keywords like “<em>user</em>”, and the system intelligently processes both approaches while providing contextual Gemini insights about the found datasets.</p><p>The implementation has two steps: first, the NLP pipeline detects question patterns, extracts meaningful terms, and provides intelligent fallback when exact matches aren’t found. Then, Gemini analyzes the actual search results to provide conversational insights that directly address the user’s query.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6889caa69065c11a640d0bec2520ff63/href">https://medium.com/media/6889caa69065c11a640d0bec2520ff63/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c83166f59a3000b291348a1f8185bf0/href">https://medium.com/media/9c83166f59a3000b291348a1f8185bf0/href</a></iframe><p>Here is the search experience:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tNzB127m2OrKiZ96gJlvNQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Dataset Cards</strong></p><p>The dataset card component presents table metadata in a clean, information-dense format optimized for the IDE. Each card displays essential information like table name, description, column count, and data types.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be6b671b379a3879695a6afd0d5f743a/href">https://medium.com/media/be6b671b379a3879695a6afd0d5f743a/href</a></iframe><p>Here is a dataset card example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CyPzIn2eKgEUiewLH8B2cQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Interactive Lineage</strong></p><p>The lineage visualization provides an interactive graph-based view of data relationships using ReactFlow, allowing users to explore upstream and downstream dependencies visually. The implementation supports drag-and-drop interaction, zoom controls, and automatic layout using the ELK (Eclipse Layout Kernel) algorithm for optimal node positioning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b1d16328b0c8ffe18f6f574e583dbcc/href">https://medium.com/media/8b1d16328b0c8ffe18f6f574e583dbcc/href</a></iframe><p>Here is the lineage experience inside Cursor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EPR8vbPjsi07vSuTjFOHUA.png" /><figcaption>Image by author</figcaption></figure><h4>Building from source</h4><p>If you want to build the extension locally for you to evolve it for your own context, you need to:</p><p><strong>Clone the repository</strong></p><p>git clone https://github.com/hugozanini/open-metadata-cursor-extension.git cd open-metadata-cursor-extension</p><p><strong>Install dependencies</strong></p><p>npm install</p><p><strong>Build the extension</strong></p><p>npm run compile</p><p><strong>Run in Development Mode</strong></p><ul><li>Open the project in VS Code/Cursor</li><li>Press F5 to launch Extension Development Host</li><li>A new window opens with the extension loaded for testing</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-UXyu2rR0cSD8KEjwF2iA.png" /><figcaption>Image by author</figcaption></figure><h4>Using the compiled version</h4><p>Download the <a href="https://github.com/hugozanini/open-metadata-cursor-extension/releases/download/1.1/openmetadata-ai-explorer-1.1.0.vsix">openmetadata-ai-explorer.vsix</a> from the <a href="https://github.com/hugozanini/open-metadata-cursor-extension/releases/tag/1.1">extension release page</a>.</p><p>Press CMD + Shift + P, type <em>Extensions: Install from VSIX... </em>and select the .vsix you downloaded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tzrGvF0SjIKpafsXgx3mzw.png" /><figcaption>Image by author</figcaption></figure><h4>Configuration setup</h4><p>Indepedent of the way you installed the extension, you need to configure it with settings of the services we are using. To do so, press Cmd+Shift+P, type: Preferences: Open User Settings (JSON), press Enter and add the following configuration parameters:</p><pre>{ &quot;openmetadataExplorer.openmetadataUrl&quot;: &quot;http://localhost:8585&quot;, <br>&quot;openmetadataExplorer.geminiApiKey&quot;: &quot;AIza_YOUR_API_KEY_HERE&quot;, <br>&quot;openmetadataExplorer.openmetadataAuthToken&quot;: &quot;YOUR_OPENMETADATA_BOT_TOKEN_HERE&quot; }</pre><p><strong>To get your Gemini API key:</strong></p><ul><li>Visit <a href="https://aistudio.google.com/">Google AI Studio</a></li><li>Sign in with Google account</li><li>Click “Get API Key” → “Create API Key”</li><li>Copy the generated key</li><li>Paste into extension settings</li></ul><p><strong>To get your OpenMetadata bot token</strong></p><ul><li>Access OpenMetadata UI: <a href="http://localhost:8585/">http://localhost:8585</a></li><li>Login with default credentials:</li><li>Navigate to: <strong>Settings</strong> → <strong>Integrations</strong> → <strong>Bots</strong></li><li>Click <strong>“Add Bot”</strong></li><li>Fill in bot details</li><li>Click <strong>“Save”</strong> → <strong>“Generate Token”</strong></li><li>Copy the generated token</li><li>Paste into extension settings</li></ul><p><strong>By now, you should have your extension up and running on Cursor :)</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fkua_mE0_ZKk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dkua_mE0_ZKk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fkua_mE0_ZKk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e276ef3a4b9dfafc053bfcf2dc2ca8d6/href">https://medium.com/media/e276ef3a4b9dfafc053bfcf2dc2ca8d6/href</a></iframe><p>This is a minimal but useful extension meant to reduce context switching and keep you in flow inside Cursor. It shows how to combine OpenMetadata and Gemini to quickly find and understand data without leaving the IDE.</p><p>Contributions are welcome in the <a href="https://github.com/hugozanini/open-metadata-cursor-extension">GitHub repository.</a> If you have questions, reach<a href="https://www.linkedin.com/in/hugozanini/"> out to me on LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=539c504a4c23" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/building-a-gemini-powered-data-catalog-extension-for-cursor-539c504a4c23">Building a Gemini-powered data catalog extension for Cursor</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Chrome extension for summarizing Google Docs comments with Gemini]]></title>
            <link>https://medium.com/data-science-collective/building-a-chrome-extension-for-summarizing-google-docs-comments-with-gemini-da14475ba91b?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/da14475ba91b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[chrome-extension]]></category>
            <category><![CDATA[google-docs]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Sun, 22 Jun 2025 02:51:24 GMT</pubDate>
            <atom:updated>2025-06-22T02:51:24.474Z</atom:updated>
            <content:encoded><![CDATA[<p>As a product manager at a fast-growing tech company, I spend a significant portion of my day navigating through Google Docs filled with comments from various stakeholders and engineers. While comments are invaluable for collaboration, they can quickly become overwhelming. A typical product spec might gather dozens or even hundreds of comments across multiple review cycles.</p><p>Finding the key points buried in lengthy comment threads became increasingly time-consuming. I&#39;d often miss important feedback or spend too much time trying to synthesize everyone&#39;s thoughts into actionable items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*XkcIwKnC_pBTQed8oibX2A.png" /><figcaption>Example of a doc I was working on last week (171 comments and counting 😬) — Image by author</figcaption></figure><p>My first attempt at solving this problem was straightforward but not very efficient: copy-pasting comment threads into AI tools like Gemini or ChatGPT. While these tools could generate summaries, the process was far from smooth:</p><ul><li>Comments would lose their <strong>formatting</strong> and context when pasted.</li><li>The <strong>manual</strong> process created significant friction.</li><li>I couldn’t easily reference the <strong>original</strong> <strong>comments</strong>.</li></ul><blockquote><strong>I also explored Google Workspace&#39;s built-in AI features, hoping to find a native solution. Unfortunately, while Google has added AI capabilities to various parts of Workspace, comment summarization wasn&#39;t among them.</strong></blockquote><h4>The decision to build a chrome extension</h4><p>After exploring available options, I decided to build a custom solution: a Chrome extension that would integrate directly with Google Docs and leverage AI to summarize comment threads on demand.</p><p>The extension, which I named ACOT (AI Comments Tool), would need to:</p><p>1. <strong>Detect and access</strong> comment threads in Google Docs</p><p>2.<strong> Send comment content to an AI</strong> service for summarization</p><p>3. Display the summaries in a <strong>convenient interface</strong></p><p>4. Support both <strong>cloud-based AI</strong> services (like Gemini API) and <strong>local</strong> options (for privacy-sensitive scenarios)</p><p>This approach would eliminate the friction of manual copying, preserve context, and provide a seamless experience right within Google Docs.</p><h3>The building blocks of a Chrome extension</h3><p>Before diving into the code, it&#39;s important to understand how Chrome extensions work. At a high level, a Chrome extension consists of five components:</p><h4>1. Manifest file (manifest.json)</h4><p>This is the heart of any Chrome extension, defining its capabilities, permissions, and components. It tells Chrome what the extension can do and what resources it needs.</p><h4><strong>2. Background scripts</strong></h4><p>These scripts run in the background, independent of any particular web page. They handle events like user clicks on the extension icon and manage the extension&#39;s state.</p><h4>3. Content scripts</h4><p>Content scripts are injected into web pages and can interact with the page&#39;s DOM. They&#39;re essential for accessing and manipulating content on websites like Google Docs.</p><h4>4. UI Elements</h4><p>Extensions can have various UI components, such as:</p><ul><li>Popup pages (appearing when clicking the extension icon)</li><li>Options pages (for settings)</li><li>Side panels (as introduced in Chrome 114)</li></ul><h4>5. Resources</h4><p>These include icons, images, and other assets used by the extension.</p><h3>Now that we understand the basics, let&#39;s build ACOT step by step</h3><p><strong>Step 1: Setting up the project structure</strong></p><p>First, let&#39;s create a directory structure for our extension:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d918502f1ecbcd7cbffd173e7da1beeb/href">https://medium.com/media/d918502f1ecbcd7cbffd173e7da1beeb/href</a></iframe><p><strong>Step 2: Creating the manifest</strong></p><p>The manifest.json file defines what our extension can do and what permissions it needs. Here&#39;s what we&#39;ll need for ACOT:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d378c9a0dc0f1f883665a87805ab38ff/href">https://medium.com/media/d378c9a0dc0f1f883665a87805ab38ff/href</a></iframe><p>This manifest:</p><ul><li>Specifies our extension’s name, version, and description</li><li>Defines icons of different sizes for various contexts</li><li>Sets up a service worker background script</li><li>Configures a side panel UI</li><li>Requests necessary permissions (side panel, storage for settings, active tab)</li><li>Specifies that our content script should run on Google Docs document pages</li><li>Requests permission to access Google Docs and the Gemini API</li></ul><p><strong>Step 3: Building the background script</strong></p><p>The background script will handle communication between the content script and the side panel, as well as API calls to the AI providers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/823fe1f0633f8474d6b8dfae7beb37af/href">https://medium.com/media/823fe1f0633f8474d6b8dfae7beb37af/href</a></iframe><p>This background script handles several important tasks:</p><ul><li>Setting up default configuration on installation</li><li>Managing communication between components</li><li>Processing API requests to AI services</li><li>Handling the extension icon click event</li><li>Testing API connections</li></ul><p><strong>Step 4: Creating the content script</strong></p><p>The content script is injected into Google Docs and interacts with the page to extract comments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dfabd60830e4f8055f066628835237f/href">https://medium.com/media/5dfabd60830e4f8055f066628835237f/href</a></iframe><p>This content script:</p><ul><li>Injects into Google Docs pages</li><li>Sets up observers to detect when comments are loaded</li><li>Adds click listeners to comments</li><li>Extracts data from comment threads</li><li>Sends comment data to the background script for processing</li></ul><p><strong>Step 5: Creating the side panel UI</strong></p><p>The side panel provides the user interface for our extension. We&#39;ll create two files: an HTML file for structure and a JavaScript file for functionality.</p><p>First, the HTML:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d3547cd8ba20c22de58d88a7162aefe/href">https://medium.com/media/5d3547cd8ba20c22de58d88a7162aefe/href</a></iframe><p>Next, the JavaScript for the side panel:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5196d26dfdd2dea757572e99cd7fc1ae/href">https://medium.com/media/5196d26dfdd2dea757572e99cd7fc1ae/href</a></iframe><p><strong>Step 6: Adding Styles</strong></p><p>Let’s create a clean interface with styles.css:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3b847ffeb74fd6e4132fe86d8045bad/href">https://medium.com/media/d3b847ffeb74fd6e4132fe86d8045bad/href</a></iframe><p>We are done!</p><p><strong>Step 7: Testing and Debugging</strong></p><p>Now that we have all the core components, let’s test our extension:</p><p>Loading the Extension:</p><ol><li>Open Chrome and navigate to chrome://extensions/</li><li>Enable “Developer mode” (toggle in the top-right corner)</li><li>Click “Load unpacked” and select your extension folder</li><li>The ACOT icon should appear in your Chrome toolbar. Click on it.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l0pYJjgjbSpitjyg1RtVEw.png" /><figcaption>Image by author</figcaption></figure><p><strong>Step 8: Setting up the LLM</strong></p><p>At this point, you should be seeing the ACOT extension opened in your chrome left panel and you can configure it to send comments to Gemini or a local endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*9bOa0W1kCugBgzRhpjJGGw.png" /><figcaption>Image by author</figcaption></figure><p>To get the Gemini API key:</p><p>Visit <a href="https://aistudio.google.com/">Google AI Studio</a></p><ol><li>Sign in with your Google account</li><li>Click “Get API key” or “Create API key”</li><li>Copy the key and paste it in ACOT’s settings</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rU2YFNyO_JKazroRw_zfXw.png" /><figcaption>Image by author</figcaption></figure><p>Select the Gemini model you want to use and click on Test Connection. If you receive a successful connection message, you can save the configuration, and you’re ready to go!</p><p><em>The free tier of Gemini API is quite generous and should handle most comment summarization needs.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*LwPgq-EE7KXEwjJqk2fTyw.png" /><figcaption>Image by author</figcaption></figure><p>For local models, you just need to define the API endpoint, test the connection, and save the configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*8tB8dePmHtSUnjTsi3aF8w.png" /><figcaption>Image by author</figcaption></figure><h3>Testing with Google Docs</h3><p>To use the tool, you just need to click on the comment and the summary will appear on the right pannel :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/0*rHnZAN8jO51v-CeU.png" /><figcaption>Image by author</figcaption></figure><h3>Lets keep building!</h3><p>ACOT is available as an <a href="https://github.com/hugozanini/acot/tree/main">open-source project on GitHub </a>under the MIT license; contributions are very welcome! The codebase serves as a practical reference for developers interested in building AI-integrated browser extensions or working with the complex DOM structure of Google Docs.</p><h4>Potential Improvements</h4><ul><li><strong>Batch processing</strong>: Implement document-wide comment analysis to generate comprehensive summary reports</li><li><strong>Context-aware summaries</strong>: Incorporate document content and comment position to provide more contextual summaries</li><li><strong>Multi-language support</strong>: Extend summarization capabilities to non-English documents and comments</li><li><strong>Advanced filtering</strong>: Allow users to filter comments by author, date range, or content type before summarization</li></ul><h3>Acknowledgments</h3><p>This project was developed during the AI sprints organized by <a href="https://developers.google.com/community/experts">Google’s Developer Expert Program</a>. Thanks, Google, for providing computing credits for making this happen and the continuous support for the open-source AI community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da14475ba91b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/building-a-chrome-extension-for-summarizing-google-docs-comments-with-gemini-da14475ba91b">Building a Chrome extension for summarizing Google Docs comments with Gemini</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Jira project management with Gemini 2.0 and Crew AI]]></title>
            <link>https://medium.com/data-science-collective/automating-jira-project-management-with-gemini-2-0-and-crew-ai-40a5b5d03372?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/40a5b5d03372</guid>
            <category><![CDATA[jira]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[crew-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Wed, 26 Feb 2025 19:08:39 GMT</pubDate>
            <atom:updated>2025-02-26T22:24:07.188Z</atom:updated>
            <content:encoded><![CDATA[<p>As a product lead managing a program that involves 15 different teams and over 80 stakeholders on a data platform, I often spend a significant amount of time gathering and processing information from various teams, understanding their work, and following up on work item status.</p><p>This work is something I don&#39;t enjoy doing, and that was cutting into my productivity and effectiveness as a leader. I haven’t had enough time to focus on the necessary product strategy for managing such a large and complex effort.</p><p>To solve this problem, I used <strong>Crew AI</strong> and <strong>Gemini 2.0</strong> to create a team of AI agents to handle my project management tasks automatically while maintaining quality and a friendly tone in communications. I named this solution TIGER:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*fDNVz8mbLjkMFlvQHW65vw.png" /><figcaption>Image by author</figcaption></figure><p>TIGER consists of a crew of AI agents that independently extract data from Jira, analyze it, and send status updates via Slack. The entire system runs in a notebook with a straightforward workflow:</p><ol><li>Extract data from Jira for a specified time period</li><li>Process and analyze the extracted data</li><li>Generate insights and follow-up questions</li><li>Consolidate the information into a comprehensive report</li><li>Send formatted updates to Slack</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezVS7A5IJ8L9L94fdxmOyA.png" /><figcaption>Image by author</figcaption></figure><p>The whole solution is open source, and this article is going to show how to setup Tiger in your company.</p><h4>The Jira Board</h4><p>The first step is to properly set up a Jira board, as this is where the agents&#39; input data will come from. I used Jira Product Discovery to create well-structured cards outlining the planned work. I also request that engineers and PMs commit to keeping the cards updated with comments and subtasks they are working on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQu1vJRhAP0vrAYQbxlygw.png" /><figcaption>Jira product board example (Image by author)</figcaption></figure><h4>Setting up the environment</h4><p>Tiger is running in a notebook. For the purpose of the tutorial, we will consider it to be executed in colab, but you can also run it locally.</p><p><strong>Configuring the API Keys</strong></p><p>You’ll need to set up several API keys and we will add them into the environment by using the Secrets tab on the colab notebook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*3jN9tfrcQQOKm_Q3OFiPOg.png" /><figcaption>Image by author</figcaption></figure><p><strong>Gemini</strong></p><p>Gemini offers a free tier for testing purposes. While it has a low rate limit, this should be enough for the purposes of this project. To get the free tier access, go to Google AI Studio and click on “Get API Key.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cv8z8v4aqqEzdQ0dA9zSrg.png" /><figcaption>Image by author</figcaption></figure><p>Then, click on Create API Key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tev8A-1kYgCMOkQcFIixkg.png" /><figcaption>Image by author</figcaption></figure><p>Select a project in your GCP account (in case you don’t have one, please create one), and then click on Create API key in the existing project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i337J9cpwdM0fHZnThdXYw.png" /><figcaption>Image by author</figcaption></figure><p>You will be prompted for the API key. Copy it and return to the colab notebook on the Secrets tab.</p><p>Click on Add new secret and create a new key with the name GEMINI_API_KEY and the Key you got in the previous step.</p><p>You should see this screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*sN8WnWS9eKaqZ26ULy6cuQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Jira API Key</strong></p><p>To get the Jira API Key, you need to follow these steps:</p><ol><li>Log in to <a href="https://id.atlassian.com/manage-profile/security/api-tokens">https://id.atlassian.com/manage-profile/security/api-tokens</a>.</li><li>Select <strong>Create API token</strong>.</li><li>Give your API token a name that describes what it does.</li><li>Select an <strong>expiration date</strong> for the API token.</li><li>Token expiration is 1 to 365 days.</li><li>Select <strong>Create</strong>.</li><li>Select <strong>Copy to clipboard</strong>, then return to the Secrets tab on the notebook and add a new key named JIRA_API_TOKEN.</li></ol><p>You also need to add a variable with the Jira URL of your company in the format <a href="https://nubank.atlassian.net/">https://&lt;your-company&gt;.atlassian.net</a> and the username you use to log on to Jira in the format<em> your.user.name@yourcompany.com.</em> Your variables setup should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*6D5A65aHOuygJuhvsf-jyQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Slack bot API Key (Optional)</strong></p><p>This part is required only if you want to create a Slack app to send the AI agent message in your company’s Slack channel. To do so, access <a href="https://api.slack.com/apps">this page</a>, click Create An App, and follow the steps to create your own app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPUTW1jVu-mq9C3JHmJxOw.png" /><figcaption>Image by author</figcaption></figure><p>Then, go to the OAuth &amp; Permissions tab, create a Bot User OAuth Token for yourself, copy the value, and return to the Colab notebook secrets tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*CJchqILG1hKoWDKnyYkE9g.png" /><figcaption>Image by author</figcaption></figure><p>Create a new environment variable called SLACK_BOT_TOKEN and paste your API Token. If you see a setup like this, you are ready to proceed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*qibQrdig-cGVPf2wK0SYEA.png" /><figcaption>Image by author</figcaption></figure><p><strong>Configuring the environment</strong></p><p>Run the cells below to clone the repo and install the required libraries. After running the code, Colab will request that you restart the session; please do so.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1335cf08507dc6e3a92d7cd14bd368c/href">https://medium.com/media/c1335cf08507dc6e3a92d7cd14bd368c/href</a></iframe><p>We will now import the required libraries and define the model to be used; our choice will be the Gemini 2.0 flash.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1eb94e0682cad4270c3019d7d59a4e5d/href">https://medium.com/media/1eb94e0682cad4270c3019d7d59a4e5d/href</a></iframe><p>Then, we will set up the information that will be used in the notebook by agents — this will change depending on your context. The example is considering a data extraction between 18 and 25 February 2025:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/341e319ec811d20dd2bdad7763715875/href">https://medium.com/media/341e319ec811d20dd2bdad7763715875/href</a></iframe><h4>Creating the agents’ crew</h4><p>The sections below show how Tiger agents, tasks, and crews were assembled. If you want to jump straight to the consolidated colab notebook, <a href="https://github.com/hugozanini/jira-tiger/blob/main/Tiger.ipynb">click here</a>.</p><p><strong>Data extraction and consolidation</strong></p><p>The first agent will be Analytics Engineer. They will have a data ingestion and data processing task that, in the end, will generate a CSV file under the jira-weekly-data folder and markdown files under the teams-markdown folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85a5b85b424a9ea5ae0172c06040bf82/href">https://medium.com/media/85a5b85b424a9ea5ae0172c06040bf82/href</a></iframe><p>Once the execution finishes, open the files under the teams-markdown to check if everything is written properly. You should see a report in the following format:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBxzKlrpY8_nRoEOqFQ_XA.png" /><figcaption>Image by author</figcaption></figure><p>If everything is right, you are ready to proceed to the next step.</p><h4>Project analysis and update generation</h4><p>This agent will be in charge of getting the reports generated by the previous agent, analyzing them to suggest follow-ups, and creating weekly status updates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7fdf27178fdf11cf901a171b94565de/href">https://medium.com/media/d7fdf27178fdf11cf901a171b94565de/href</a></iframe><p>After the execution, you will see two JSON files consolidating the analysis. The first contains all issues in the given period and the respective follow-ups and updates, as shown in the snippet below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f00b054b4467fc4506cf090af8d27f6/href">https://medium.com/media/5f00b054b4467fc4506cf090af8d27f6/href</a></iframe><p>The second is the report_summary.json — containing the content of the placeholders set in the previous file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6733469beeb224f97d6c86d7be25e43d/href">https://medium.com/media/6733469beeb224f97d6c86d7be25e43d/href</a></iframe><h4>Sending the Slack message</h4><p>With all the follow-ups and status updates, the communication specialist is ready to get the data and send the message on the slack channel I have with my stakeholders:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6daa3271e595c4612dd7b3036f927907/href">https://medium.com/media/6daa3271e595c4612dd7b3036f927907/href</a></iframe><p>A real example of a message sent by the communication specialist agent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*liAyCt0CjpBGG6hH" /><figcaption>Image by author</figcaption></figure><h4>Focusing on what matters</h4><p>As I said in the beginning, the agents are saving me a lot of time — at least 5 hours per week, and I have much more space to focus on engaging in product strategy discussions with my stakeholders.</p><p>As a good product manager, I ran some surveys with the more than 80 people involved in the initiative, and here are some numbers about their experience with TIGER:</p><ul><li>80% evaluated Tiger updates as Good or Great</li><li>80% said Tiger is much better than the previous synchronous weekly updates</li><li>90% said they would be very disappointed or somewhat disappointed if we stop to use Tiger</li><li>66% said Tiger somehow saved time when providing work items updates</li><li>80% said they would be committed to providing weekly updates on their work items on Jira</li></ul><p>I hope this article inspires you to use AI Agents in your daily work as a product manager. Feel free to reach out to me on <a href="https://www.linkedin.com/in/hugozanini/">LinkedIn</a> if you have questions.</p><p>The complete implementation is available on GitHub: <a href="https://github.com/hugozanini/jira-tiger/blob/main/Tiger.ipynb">github.com/hugozanini/jira-tiger</a></p><h4>Acknowledgments</h4><p>This project was developed during the Vertex sprints organized by Google’s Developer Expert Program. Thanks, Google, for providing GCP and Colab credits for making this happen and the continuous support for the open-source AI community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40a5b5d03372" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/automating-jira-project-management-with-gemini-2-0-and-crew-ai-40a5b5d03372">Automating Jira project management with Gemini 2.0 and Crew AI</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a User Insights-Gathering Tool for Product Managers from Scratch]]></title>
            <link>https://medium.com/data-science/building-a-user-insights-gathering-tool-for-product-managers-from-scratch-a6459dc1c3f6?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6459dc1c3f6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[gemma]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 15:57:26 GMT</pubDate>
            <atom:updated>2024-08-12T15:57:26.513Z</atom:updated>
            <content:encoded><![CDATA[<h4>Say goodbye to paid customer insights hubs! Learn how to combine five open-source AI models to automate insights gathering from your user interviews.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CpbiyAxQEBlH72z6-8gzIQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@epicantus?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Daria Nepriakhina </a>on <a href="https://unsplash.com/photos/printed-sticky-notes-glued-on-board-zoCDWPuiRuA?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>As a technical product manager for a data platform, I frequently run user interviews to identify challenges associated with data development processes.</p><p>However, when exploring a new problem area with users, I can easily become overwhelmed by the numerous conversations I have with various individuals across the organization.</p><p>Over time, I have adopted a systematic approach to address this challenge. I focus on taking comprehensive notes during each interview and then revisit them. This allows me to consolidate my understanding and identify user discussion patterns.</p><p>However, dividing my attention between note-taking and active listening often compromised the quality of my conversations. I noticed that when someone else took notes for me, my interviews significantly improved. This allowed me to fully engage with the interviewees, concentrate solely on what they were saying, and have more meaningful and productive interactions.</p><p>To be more efficient, I transitioned from taking notes during meetings to recording and transcribing them whenever the functionality was available. This significantly reduced the number of interviews I needed to conduct, as I could gain more insights from fewer conversations. However, this change required me to invest time reviewing transcriptions and watching videos. The figure below shows what became a simplified flow of the process I follow for mapping a new product development opportunity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kKwObiHU9U_aPY67" /><figcaption>Image by author</figcaption></figure><p><strong>Due to the size of the meeting transcriptions and difficulties in categorizing user insights, consolidation and analysis became challenging.</strong></p><p>Furthermore, the meeting transcription tools available to me were limited to English, while most of my conversations were in Portuguese. As a result, I decided to look to the market for a solution that could help me with those challenges.</p><p>The solutions I found that solved most of my pain points were <a href="https://dovetail.com/">Dovetail</a>, <a href="https://heymarvin.com/">Marvin</a>, <a href="https://condens.io/">Condens</a>, and <a href="https://reduct.video/">Reduct</a>. They position themselves as customer insights hubs, and their main product is generally Customer Interview transcriptions.</p><p>Basically, you can upload an interview video there, and you are going to receive a transcription indicating who is speaking with hyperlinks to the original video on every phrase. Over the text, you can add highlights, tags, and comments and ask for a summary of the conversation. <strong>These features would solve my problem; however, these tools are expensive, especially considering that I live in Brazil and they charge in dollars.</strong></p><blockquote><strong>The tools offer nothing revolutionary, so I decided to implement an open-source alternative that would run for free on Colab notebooks.</strong></blockquote><h4>The requirements</h4><p>As a good PM, the first thing I did was identify the must-haves for my product based on the user&#39;s (my) needs. Here are the high-level requirements I mapped:</p><p><strong>Cost and Accessibility</strong></p><ul><li>Free;</li><li>No coding experience is required to use;</li></ul><p><strong>Data Privacy and Security</strong></p><ul><li>Keep data private — no connection with external services;</li></ul><p><strong>Performance</strong></p><ul><li>Execution must be faster than the video duration;</li><li>High-precision transcription in different languages;</li></ul><p><strong>Functionality</strong></p><ul><li>Speakers identification;</li><li>Easy to search over the transcriptions;</li><li>Easy to highlight the transcriptions;</li><li>Easy to create repositories of research being done;</li></ul><p><strong>Integration</strong></p><ul><li>Integrated with my company&#39;s existing tools (Google Workspace);</li><li>LLM model integrated to receive prompts for tasks on top of the transcriptions;</li></ul><h4>The solution</h4><p>With the requirements, I designed what would be the features of my solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9KI77PhDCu_7OHX6JFkwg.png" /><figcaption>Image by author</figcaption></figure><p>Then, I projected what would be the expected inputs and the user interfaces to define them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OX_Efh1DoQWCZLnFrYHtUA.png" /><figcaption>Image by author</figcaption></figure><p>Users will upload their interview to YouTube as an unlisted video and create a Google Drive folder to store the transcription. They will then access a Google Colab notebook to provide basic information about the interview, paste the video URL, and optionally define tasks for an LLM model. The output will be Google Docs, where they can consolidate insights.</p><p>Below is the product architecture. The solution required combining five different ML models and some Python libraries. The next sections provide overviews of each of the building blocks; however, if you are more interested in trying the product, please go to the &quot;I got it&quot; section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zMmv0aPtsPLM_Q8muL3MKQ.png" /><figcaption>Image by author</figcaption></figure><h4>Interview Setup and Video Upload</h4><p>To create a user-friendly interface for setting up interviews and providing video links, I used <a href="https://colab.research.google.com/notebooks/forms.ipynb?ref=dataschool.io#scrollTo=ig8PIYeLtM8g">Google Colab’s forms functionality</a>. This allows for the creation of text fields, sliders, dropdowns, and more. The code is hidden behind the form, making it very accessible for non-technical users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QupHpXm_eyx8Q-2TIb8qmA.png" /><figcaption>Interview selection form — Image by author</figcaption></figure><h4>Audio download and conversion</h4><p>I used the yt-dlp lib to download only the audio from a YouTube video and convert it to the mp3 format. It is very straightforward to use, and you can <a href="https://github.com/yt-dlp/yt-dlp">check its documentation here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*adZivV_QtcMgStHTXShnAA.png" /><figcaption>Image by <a href="https://github.com/yt-dlp/yt-dlp">yt-dlp</a></figcaption></figure><h4>Audio transcription</h4><p>To transcribe the meeting, I used <a href="https://github.com/openai/whisper">Whisper from Open AI</a>. It is an open-source model for speech recognition trained on more than 680K hours of multilingual data.</p><p>The model runs incredibly fast; a one-hour audio clip takes around 6 minutes to be transcribed on a 16GB T4 GPU (offered by free on Google Colab), and it supports <a href="https://github.com/openai/whisper/blob/ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab/whisper/tokenizer.py#L11-L110">99 different languages</a>.</p><p>Since privacy is a requirement for the solution, the model weights are downloaded, and all the inference occurs inside the colab instance. I also added a Model Selection form in the notebook so the user can choose different models based on the precision they are looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ZawmIEwy1mMA16V5izj9HQ.png" /><figcaption>Image by author</figcaption></figure><h4>Speakers Identification</h4><p>Speaker identification is done through a technique called Speakers Diarization. The idea is to identify and segment audio into distinct speech segments, where each segment corresponds to a particular speaker. With that, we can identify who spoke and when.</p><p>Since the videos uploaded from YouTube don&#39;t have metadata identifying who is speaking, the speakers will be divided into Speaker 1, Speaker 2, etc.… Later, the user can find and replace those names in Google Docs to add the speakers’ identification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fr086sIOgyvdBd775xXhqA.png" /><figcaption>Image by author</figcaption></figure><p>For the diarization, we will use a model called the <a href="https://arxiv.org/pdf/2203.15974">Multi-Scale Diarization Decoder (MSDD)</a>, which was developed by Nvidia researchers. It is a sophisticated approach to speaker diarization that leverages multi-scale analysis and dynamic weighting to achieve high accuracy and flexibility.</p><p>The model is known for being quite good at identifying and properly categorizing moments where multiple speakers are talking—a thing that occurs frequently during interviews.</p><p>The model can be used through the <a href="https://github.com/NVIDIA/NeMo">NVIDIA NeMo framework</a>. It allowed me to get MSDD checkpoints and run the diarization directly in the colab notebook with just a few lines of code.</p><p>Looking into the Diarization results from MSDD, I noticed that punctuation was pretty bad, with long phrases, and some interruptions such as &quot;<em>hmm</em>&quot; and &quot;<em>yeah</em>&quot; were taken into account as a speaker interruption — making the text difficult to read.</p><p>So, I decided to add a punctuation model to the pipeline to improve the readability of the transcribed text and facilitate human analysis. So I got the <a href="https://huggingface.co/kredor/punctuate-all">punctuate-all model from Hugging Face</a>, which is a very precise and fast solution and supports the following languages: English, German, French, Spanish, Bulgarian, Italian, Polish, Dutch, Czech, Portuguese, Slovak, and Slovenian.</p><h4>Video Synchronization</h4><p>From the industry solutions I benchmarked, a strong requirement was that every phrase should be linked to the moment in the interview the speaker was talking.</p><p>The Whisper transcriptions have metadata indicating the timestamps when the phrases were said; however, this metadata is not very precise.</p><p>Therefore, I used a model called <a href="https://arxiv.org/abs/2006.11477">Wav2Vec2</a> to do this match in a more accurate way. Basically, the solution is a neural network designed to learn representations of audio and perform speech recognition alignment. The process involves finding the exact timestamps in the audio signal where each segment was spoken and aligning the text accordingly.</p><p>With the transcription &lt;&gt; timestamp match properly done, through simple Python code, I created hyperlinks pointing to the moment in the video where the phrases start to be said.</p><h4>The LLM Model</h4><p>This step of the pipeline has a large language model ready to run locally and analyze the text, providing insights about the interview. By default, I added a Gemma Model 1.1b with a prompt to summarize the text. If the users choose to have the summarization, it will be in a bullet list at the top of the document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INx6G3y6wLsO447haKo-nQ.png" /><figcaption>Image by author</figcaption></figure><p>Also, by clicking on <em>Show code</em>, the users can change the prompt and ask the model to perform a different task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*jaPcJt4OCHq8uLe6QFjteg.png" /></figure><h4>Document generation for tagging, highlights, and comments</h4><p>The last task performed by the solution is to generate Google Docs with the transcriptions and hyperlinks to the interviews. This was done through the <a href="https://googleapis.github.io/google-api-python-client/">Google API Python client library</a>.</p><h3>I got it</h3><p>Since the product has become incredibly useful in my day-to-day work, I decided to give it a name for easier reference. I called it the <strong>I</strong>nsights <strong>G</strong>athering <strong>O</strong>pen-source <strong>T</strong>ool, or iGot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GWV6oLY2bXPLj0dJBtn54w.png" /><figcaption><a href="https://openai.com/policies/terms-of-use/">Image generated by DALL·E-3</a></figcaption></figure><p>When using the solution for the first time, some initial setup is required. Let me guide you through a real-world example to help you get started.</p><h4>Open the iGot notebook and install the required libraries</h4><p>Click on <a href="https://colab.research.google.com/drive/1eMN4-b4kAH4uNTzzgJ-X4J90BYozGl6o?usp=sharing">this link</a> to open the notebook and run the first cell to install the required libraries. It is going to take around 5 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PztJds_tBrcY1wSBig7_xA.png" /><figcaption>Image by author</figcaption></figure><p>If you get a prompt asking you to restart the notebook, just cancel it. There is no need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*t4FRn9aIViVGgF1k9GROdw.png" /><figcaption>Image by author</figcaption></figure><p>If everything runs as expected, you are going to get the message &quot;All libraries installed!&quot;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MbYlKVW9Cjq6B7tdFh2-6g.png" /><figcaption>Image by author</figcaption></figure><h4>Getting the Hugging User Access Token and model access</h4><p><em>(This step is required just the first time you are executing the notebook)</em></p><p>For running the Gemma and punctuate-all models, we will download weights from hugging face. To do so, you must request a user token and model access.</p><p>To do so, you need to create a hugging face account and follow these steps to get a token with reading permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*OYe4yE3tvD4Ki2srT4OGag.png" /><figcaption>Image by author</figcaption></figure><p>Once you have the token, copy it and return to the lab notebook. Go to the secrets tab and click on &quot;Add new secret.&quot;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*duRbv5L4P6MUYJ5c5xeUVQ.png" /><figcaption>Image by author</figcaption></figure><p>Name your token as <em>HF_TOKEN</em> and past the key you got from Hugging Face.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*uhA7R0Yfll1dJhjhbZOEHA.png" /><figcaption>Image by author</figcaption></figure><p>Next, click <a href="https://medium.com/r?url=https%3A%2F%2Fhuggingface.co%2Fgoogle%2Fgemma-1.1-2b-it">this link</a> to open the Gemma model on Hugging Face. Then, click on “Acknowledge license” to get access the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*WBuIlknRN7iB0bdA6wRk-w.png" /><figcaption>Image by author</figcaption></figure><h4>Sending the interview</h4><p>To send an interview to iGot, you need to upload it as an unlisted video on YouTube previously. For the purpose of this tutorial, I got a piece of the Andrej Karpathy interview with Lex Fridman and uploaded it to my account. It is part of the conversation where Andrej gave some advice for Machine Learning Beginners.</p><p>Then, you need to get the video URL, paste in the <em>video_url</em> field of the <em>Interview Selection</em> notebook cell, define a name for it, and indicate the language spoken in the video.</p><p>Once you run the cell, you are going to receive a message indicating that an audio file was generated.t into</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RfFzQ7XhG9-6EDbRLOZXoA.png" /><figcaption>Image by author</figcaption></figure><h4>Model selection and execution</h4><p>In the next cell, you can select the size of the Whisper model you want to use for the transcription. The bigger the model, the higher the transcription precision.</p><p>By default, the largest model is selected. Make your choice and run the cell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9QFoVeNiNFOf3FL7IOzHsQ.png" /><figcaption>Image by author</figcaption></figure><p>Then, run the models execution cell to run the pipeline of models showed in the previous section. If everything goes as expected, you should receive the message &quot;Punctuation done!&quot; by the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zwQW53OqGj8S1Wx2A_o3jw.png" /><figcaption>Image by author</figcaption></figure><p>If you get prompted with a message asking for access to the hugging face token, grant access to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*TO7Oq_70pLWb3OV3Ccdcog.png" /><figcaption>Image by author</figcaption></figure><h4>Configuring the transcript output</h4><p>The final step is to save the transcription to a Google Docs file. To accomplish this, you need to specify the file path, provide the interview name, and indicate whether you want Gemma to summarize the meeting.</p><p>When executing the cell for the first time, you can get prompted with a message asking for access to your Google Drive. Click in allow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*xzzv35J4uBFFOZNPZhoxXg.png" /><figcaption>Image by author</figcaption></figure><p>Then, give Colab full access to your Google Drive workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*vDU7AjzPP8_MGjBUIVxWWw.png" /><figcaption>Image by author</figcaption></figure><p>If everything runs as expected, you are going to see a link to the google docs file at the end. Just click on it, and you are going to have access to your interview transcription.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hn5q7HMKR8lnx14tOp_GTQ.png" /><figcaption>Image by author</figcaption></figure><h4>Gathering insights from the generated document</h4><p>The final document will have the transcriptions, with each phrase linked to the corresponding moment in the video where it begins. Since YouTube does not provide speaker metadata, I recommend using Google Docs’ find and replace tool to substitute “Speaker 0,” “Speaker 1,” and so on with the actual names of the speakers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*j6qtRjdsp9Xu2IGHuwebtg.png" /><figcaption>Image by author</figcaption></figure><p>With that, you can work on highlights, notes, reactions, etc. As envisioned in the beginning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*_FEAfmNndse2EGbjzTwxwA.png" /><figcaption>Image by author</figcaption></figure><h3>Final thoughts</h3><p>The tool is just in its first version, and I plan to evolve it into a more user-friendly solution. Maybe hosting a website so users don’t need to interact directly with the notebook, or creating a plugin for using it in Google Meets and Zoom.</p><p>My main goal with this project was to create a high-quality meeting transcription tool that can be beneficial to others while demonstrating how available open-source tools can match the capabilities of commercial solutions.</p><p>I hope you find it useful! Feel free to <a href="https://www.linkedin.com/in/hugozanini/">reach out to me on LinkedIn</a> if you have any feedback or are interested in collaborating on the evolution of iGot :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6459dc1c3f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-user-insights-gathering-tool-for-product-managers-from-scratch-a6459dc1c3f6">Building a User Insights-Gathering Tool for Product Managers from Scratch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Desmistificando o fine tuning de LLMs na prática: PEFT, LoRA, QLORA e Hambúrgueres]]></title>
            <link>https://medium.com/data-hackers/desmistificando-o-fine-tuning-de-llms-na-pr%C3%A1tica-peft-lora-qlora-e-hamb%C3%BArgueres-ca6e6008241f?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca6e6008241f</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[artist]]></category>
            <category><![CDATA[data-scraping]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Mon, 13 May 2024 12:14:25 GMT</pubDate>
            <atom:updated>2024-05-13T12:14:25.543Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*DgRE54D-n5yBxXrz" /><figcaption>Imagem gerada pelo DALL·E 3</figcaption></figure><p>A possibilidade de utilização de grandes modelos de linguagem (LLMs) para a realização de tarefas que podem gerar ganhos de produtividade tem despertado o interesse de diversas indústrias. Recentemente, o CEO da <a href="https://www.tcs.com/">TCS</a>, maior provedora de serviços e produtos de TI no mundo, disse acreditar que, por conta avanços em AI, <a href="https://www.ft.com/content/149681f0-ea71-42b0-b85b-86073354fb73">as operações de call centers devem se tornar obsoletas em menos de um ano</a>.</p><p>Em sua entrevista, <a href="https://www.linkedin.com/in/k-krithivasan-84659325/">K Krithivasan</a> diz acreditar que em breve chatbots serão capazes de analisar dados de clientes e fazer a maioria do trabalho feito por um call center hoje — respondendo solicitações e até antecipando problemas.</p><p>Eu acredito neste futuro. Agentes de AI já são capazes de realizar boa parte das tarefas repetitivas de um call-center, mas a indústria ainda esbarra em alguns desafios.</p><p>O principal deles relaciona-se com o desenvolvimento de modelos. Atualmente, as soluções mais poderosas são fechadas e encontram-se nas mãos de empresas como OpenAI, Anthropic e Google.</p><p>Treinar um LLM do zero requer muito investimento em engenharia, GPUs, energia elétrica e coleta de dados. São poucas as organizações que têm capacidade de fazer tais investimentos. Portanto, para criar soluções, empresas menores acabam indo por dois caminhos: Pagam pela utilização soluções fechadas como o <a href="https://openai.com/gpt-4">GPT-4</a> ou <a href="https://blog.google/intl/pt-br/novidades/tecnologia/apresentando-o-gemini-nosso-maior-e-mais-habil-modelo-de-ia/#mensagem-sundar">Gemini</a>, ou tentam adaptar e evoluir modelos open-source.</p><p>Meta e Google têm liderado o movimento de disponibilização de tais modelos — só para o treino do <a href="https://llama.meta.com/llama2/">Llama2</a>, por exemplo, estima-se que a meta investiu aproximadamente <a href="https://www.livemint.com/ai/artificial-intelligence/metas-llama-2-why-open-source-llms-are-the-joker-in-the-generative-ai-pack-11689756448782.html">20 milhões de dólares</a>. Estas soluções apresentam limitações quando comparadas com os modelos fechados, mas funcionam muito bem o objetivo é utilizá-las como base para trabalhar em tarefas mais específicas.</p><p>Voltando ao exemplo do call center, se quisermos criar um chatbot para responder perguntas relacionadas a um produto especifico e para um perfil de clientes pré-determinado, a utilização de um modelo open-source como base é um caminho muito interessante.</p><p>Com poucas amostras de interações entre atendentes e clientes seria possível adaptar um modelo para adotar um estilo de comunicação e absorver informações para a resolução de problemas e execução de tarefas. Este tipo de abordagem é conhecida como fine-tuning e é o que iremos explorar neste post.</p><h4>Definindo um pipeline de fine-tuning</h4><p>A ideia básica por trás de um fine tuning é aproveitar o conhecimento e capacidade de generalização de grandes modelos e adaptar a uma tarefa ou domínio específico. O diagrama a seguir mostrar um pipeline genérico de treinamento.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*01CBXn3aag1rhx3D4nEGHA.png" /></figure><p>O <em>modelo base</em> é pré-treinado em grandes volumes de dados de texto para prever a próxima palavra em uma sentença. Estes modelos são genéricos - não possuem conhecimento profundo em um determinado domínio ou tarefa.</p><p>Um <em>modelo fine-tuned</em> é construído a partir de um treinamento adicional em cima de um <em>modelo base</em>, para executar uma tarefa específica ou absorver conhecimento sobre um domínio. Os casos de uso geralmente envolvem datasets de classificação de texto, perguntas e respostas e tradução.</p><p>A ideia central do processo de fine tuning é aproveitar o conhecimento presente nos modelos base e suas capacidades de generalização. Como os modelos já sabem realizar múltiplas tarefas, o processo de especialização se torna mais fácil - com pequenos volumes de dados é possível obter resultados razoáveis.</p><p>Durante um fine-tuning, os pesos e parâmetros do <em>modelo base </em>são ajustados a partir do dataset de treino. Neste processo, as camadas iniciais do modelo, as quais capturam features mais genéricas, geralmente se mantém congeladas. Isso faz com que o modelo mantenha seu conhecimento anterior e se adapte as novas tasks.</p><p>Atualmente, o método mais comum para fazer o fine tuning de modelos é Parameter Efficient Fine-Tuning (PEFT).</p><h4>Parameter Efficient Fine-Tuning</h4><p>O PEFT é um método projetado para ajustar grandes modelos sem a necessidade de retreinar todos os seus parâmetros, que geralmente estão na ordem de grandeza de bilhões.</p><p>Nesta abordagem, não é necessário grandes volumes de dados e muitos recursos computacionais. O PEFT faz um congelamento dos parâmetros de treino do modelo base e adiciona novos parâmetros que serão treinados em um conjunto de dados nunca visto.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TknUDNbwqYUHxUi3BsmOvQ.png" /></figure><p>Neste tipo de abordagem, prevenimos situações como um esquecimento de todas as informações que o modelo absorveu durante seu treinamento anterior. Essa é uma ótima escolha quando estamos lidando com tarefas relacionadas a um domínio específico, como o caso do call center que mencionei no início.</p><p>O PEFT utiliza técnicas como LoRa e Q-Lora para fazer o fine tuning dos parâmetros. A seguir, uma breve explicação de ambas :)</p><h4>LoRa</h4><p>O <strong>Lo</strong>w <strong>R</strong>ank <strong>A</strong>daption é um método em que adicionamos pequenos módulos de adaptação no modelo base e então treinamos somente os parâmetros destes módulos.</p><p>Na arquitetura de transformers, presentes nas LLMs, temos grandes módulos lineares, chamados de attention vectors e feed foward layers. Ambos, possuem um grande número de parâmetros e podemos escolher em qual camada adicionar os módulos LoRa que serão adaptados durante o fine tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Kbf38V10BmI6jt9aWlCVw.png" /></figure><p>Os módulos lineares do modelo base são grandes matrizes e os adaptadores LoRa são matrizes pequenas capazes de receber a mesma entrada e produzir uma saída compatível com a arquitetura existente. Como você já deve esperar, a computação dos parâmetros destes adaptadores é muito mais rápida do que as dos módulos originais do modelo.</p><h4>QLoRa</h4><p>Quantized Low Rank Adaptation, QLoRa é uma extensão do LoRa que adiciona uma camada de quantização no processamento dos parâmetros. Basicamente, reduzimos a quantidade de bits disponíveis para a armazenar os números que representam os parâmetros.</p><p>Por exemplo, imagine que tenhamos um parâmetro com o valor 128.64 e gostaríamos de quantizá-lo para 8 bits (podendo representar apenas 8 dígitos de informação). Neste caso, os passos seriam os seguintes:</p><ol><li>Converter o número de ponto flutuante para binário: 128 em binário é 10000000 e 0.64 em binário é 0.101.</li><li>Descartar os números que excedem os 8 dígitos: 128 em binário já consome todos os dígitos disponíveis, portanto, qualquer valor que vier depois dele será descartado. No nosso caso, 0.101 — que corresponde a 0.64.</li></ol><p>Dessa forma, a versão quantizada de 128.64 em um sistema de 8 bits seria 128.</p><p>Com esse tipo de operação, o modelo torna-se mais compacto e as operações mais rápidas. Consequentemente, os resultados ficam mais imprecisos, mas a capacidade de generalização do modelo pode melhorar 🤔. É um caso típico em que a experimentação é necessária.</p><p><a href="https://arxiv.org/pdf/2305.14314">Em seus benchmarkings</a>, o QLoRa mostrou ser capaz em reduzir de 780GB para 48GB a memória de GPU necessária para fazer o fine tuning de um modelo de linguagem de 65 bilhões de parâmetros. Neste caso, sem nenhuma perda mensurável de performance comparada com a versão com precisão completa dos parâmetros.</p><h4>Como isso tudo funciona na prática?</h4><p>Para aplicar todos estes conceitos, vamos realizar o fine tuning do <a href="https://blog.google/technology/developers/gemma-open-models/">Gemma 7B</a>, uma LLM disponibilizada gratuitamente pelo Google. Ela será o <em>base model </em>citado anteriormente.</p><p><em>Nota: Você pode utilizar qualquer outro modelo como Llama, Mixtral e etc. A única modificação será no nome dos módulos em que você irá inserir os adaptadores LoRa.</em></p><p>Como base de dados, iremos utilizar tweets do <a href="https://twitter.com/BurgerKingBR">Burger King Brasil</a> 🍔. A conta deles tem um estilo próprio ao realizar publicações e responder clientes, o nosso objetivo será ajustar o Gemma para que ele responda a mensagens de usuários como se fosse o administrador da conta da empresa no X.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*riisLMtvlJXDDtJCXFgIvg.png" /><figcaption>Exemplo de interação no X do Burger King Brasil</figcaption></figure><p>Para isso, extraí 468 interações da conta da empresa e usei o Gemini Pro para me ajudar a anonimizar os dados e definir quais fatos que geraram a interação.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5g7Qhu06ei638kzfoobI-A.png" /><figcaption>Amostra dos dados sendo utilizados no treinamento</figcaption></figure><p>Para o treinamento, utilizei uma Tesla T4, que é disponibilizada gratuitamente no Google Colab. A seguir vou caminhar por cada etapa do processo de preparação dos dados, definição dos adaptadores com Lora, fine tuning, inferência e disponibilização do modelo no Hugging Face 🤗. No entanto, se você quiser pular direto para o notebook, <a href="https://colab.research.google.com/drive/1tkOKo7GTvgOr6LdASjO-2SlJE17XzlDg?usp=sharing">clique aqui</a>.</p><h4>Configurando o ambiente</h4><p>Vamos começar instalando e importando as bibliotecas necessárias e fazendo login no Hugging Face, que é de onde vamos baixar o Gemma 7B e posteriormente fazer o upload do nosso modelo treinado. Para isso, <a href="https://huggingface.co/docs/hub/security-tokens">siga estas etapas</a> para gerar seu próprio token.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc218810ca95ae11d50ae597369f0c9f/href">https://medium.com/media/dc218810ca95ae11d50ae597369f0c9f/href</a></iframe><h4>Preparando os dados</h4><p>Nesta etapa, vamos carregar os dados, fazer algumas transformações e converter para o formato aceito pelo Hugging Face.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e7fb8e039bf3126ae4341f441e66b62/href">https://medium.com/media/7e7fb8e039bf3126ae4341f441e66b62/href</a></iframe><h4>Baixando e testando o modelo base</h4><p>Vamos baixar o gemma-7b via hugging face. Para isso, é necessário que você acesse <a href="https://huggingface.co/google/gemma-7b-it">este link</a> e aceite a licença do modelo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfxbxWzHDepCVi-48arFaA.png" /></figure><p>Com a licença aceita no hugging face, vamos utilizar a biblioteca <a href="https://pypi.org/project/bitsandbytes/">bitsandbites</a> para carregar o modelo com quantização de 4-bits, o que significa metade da precisão original para as computações. Como mencionado anteriormente, esta modificação fará com que seja necessário menos recursos computacionais durante o fine tuning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a0f6c470d83d51b65dcf4edc8a46563/href">https://medium.com/media/4a0f6c470d83d51b65dcf4edc8a46563/href</a></iframe><p>Com o modelo carregado, vamos fazer um teste com o mesmo prompt que será utilizado depois do fine tuning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cd84ab9c06f52a453e0fc1eb0a1d674/href">https://medium.com/media/5cd84ab9c06f52a453e0fc1eb0a1d674/href</a></iframe><h4>Definindo os parâmetros de treino e começando o fine tuning</h4><p>A primeira célula do código abaixo realiza a configuração do adaptador LoRa. Para o nosso exemplo, vamos inserir adaptadores logo após todas as camadas lineares do Gemma, definidas pelo parâmetro <em>taget_modules</em>.</p><p>O valor de <em>r</em> está relacionado com a quantidade de parâmetros que serão adaptados durante o treinamento. O <em>task_type</em> indica pra qual tarefa o modelo está sendo adaptado. Neste exemplo, estaremos trabalhando com geração de texto, então escolhemos a opção de Causal Language Model.</p><p>Para saber mais sobre os parâmetros do LoRA, recomendo a leitura do <a href="https://arxiv.org/abs/2106.09685">paper do método</a>.</p><p>Iremos utilizar o método de supervised fine-tuning (SFT) disponibilizado pelo Hugging Face para configurar o treinamento. Ele implementa o PEFT, que mencionamos anteriormente. Para saber mais sobre os parâmetros escolhidos, recomendo acessar <a href="https://huggingface.co/docs/trl/sft_trainer">esta documentação do hugging face</a>.</p><p>A última célula do notebook é onde o treinamento é iniciado. Usando uma Tesla T4 no Colab, o execução durou aproximadamente três horas. Este tempo vai variar de acordo com os parâmetros escolhidos e o o volume dados sendo utilizado.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afdd6fb7829b8d5c2fd559fd0f3ca597/href">https://medium.com/media/afdd6fb7829b8d5c2fd559fd0f3ca597/href</a></iframe><p>Os pesos treinados foram salvos na pasta <em>gemma-tunned</em>. Vamos pegar o último checkpoint disponível, fazer a junção dos adaptadores LoRa treinados com o modelo base (gemma-7b),e salvar na pasta <em>gemma-7b-tuned-merged</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afdd6fb7829b8d5c2fd559fd0f3ca597/href">https://medium.com/media/afdd6fb7829b8d5c2fd559fd0f3ca597/href</a></iframe><p><strong>Pronto! Já temos o nosso próprio fine tuning do gemma-7b :)</strong></p><p>Agora, vamos fazer um teste com o mesmo prompt que usamos com o modelo antes do fine-tuning para checar os resultados.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83e9af9757d618f8a63d326820d090a1/href">https://medium.com/media/83e9af9757d618f8a63d326820d090a1/href</a></iframe><p>Claramente, o modelo desenvolveu a capacidade de receber um fato e gerar um tweet com o estilo de comunicação do Burger King 🍔 !</p><p>Para não deixar o post ̶a̶i̶n̶d̶a̶ ̶m̶a̶i̶s̶ longo, vamos seguir para a última etapa do tutorial: salvar o modelo e o tokenizador em um repositório no hugging face hub para facilitar a reutilização.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/762f0adebeb3c803c592de8a0bdc1786/href">https://medium.com/media/762f0adebeb3c803c592de8a0bdc1786/href</a></iframe><p>Por fim, vamos carregar o modelo diretamente do hugging face hub e executar um teste novamente para garantir que tudo está funcionando bem.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99aa72eff2dd86887ae09023ba68c46d/href">https://medium.com/media/99aa72eff2dd86887ae09023ba68c46d/href</a></iframe><p>Tudo certo! Tivemos mais uma resposta satisfatória com o modelo que simula o comportamento do administrador do X do Burger King.</p><p>Espero que você tenha curtido este post e que ela tenha sido útil. Se gostou, não se esqueça de adicionar algumas 👏 aqui no medium. Para qualquer dúvida/sugestão, adicione um comentário aqui, ou me chame no <a href="https://www.linkedin.com/in/hugozanini/">Likedin </a>:)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca6e6008241f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-hackers/desmistificando-o-fine-tuning-de-llms-na-pr%C3%A1tica-peft-lora-qlora-e-hamb%C3%BArgueres-ca6e6008241f">Desmistificando o fine tuning de LLMs na prática: PEFT, LoRA, QLORA e Hambúrgueres</a> was originally published in <a href="https://medium.com/data-hackers">Data Hackers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training a custom YOLOv7 in PyTorch and running it directly in the browser with TensorFlow.js]]></title>
            <link>https://medium.com/data-science/training-a-custom-yolov7-in-pytorch-and-running-it-directly-in-the-browser-with-tensorflow-js-96a5ecd7a530?source=rss-8f4c63386bac------2</link>
            <guid isPermaLink="false">https://medium.com/p/96a5ecd7a530</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflowjs]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Hugo Zanini]]></dc:creator>
            <pubDate>Tue, 28 Mar 2023 18:38:24 GMT</pubDate>
            <atom:updated>2023-03-28T18:38:24.997Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*exIL8hVbVfWLz7QXqBXNoA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@martijnbaudoin?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Martijn Baudoin</a> on <a href="https://unsplash.com/photos/4z0-2mQE7io?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Custom YOLOv7 Object Detection with TensorFlow.js</h3><h4>Training a custom YOLOv7 model in PyTorch and converting it to TensorFlow.js for real-time offline detection on the browser</h4><p>Recently, I open-sourced an implementation of <a href="https://github.com/hugozanini/yolov7-tfjs?linkId=8607585"><strong>YOLOv7 in Tensorflow.js</strong></a> and the most common question I received was:</p><blockquote>How did you <strong>convert</strong> the model from PyTorch to Tensorflow<strong>.js</strong>?</blockquote><p>This post will cover this process by solving a real-world problem using a custom <strong>YOLOv7</strong> model running <strong>directly on the browser and offline</strong>.</p><p>The industry we will tackle is <strong>physical retail</strong>. Besides the digitalization that happened recently — mostly during the pandemic — <a href="https://www.forbes.com/sites/sap/2022/02/07/retail-trends-2022-in-search-of-the-ultimate-customer-experience/?sh=13a0809c52ad">the physical store remains the customers&#39; most preferred shopping destination</a>.</p><p>Everything in a store is about the experience. The<a href="https://feedbackgroup.com/"> Retail Feedback Group</a> (RFG), which has been tracking the grocery shopping experience for around 15 years, <a href="https://www.winsightgrocerybusiness.com/retailers/supermarket-experience-its-time-integrate">consistently finds</a> that the most crucial factor affecting customer satisfaction is whether shoppers are able to <strong>find everything they need during their visit, whether it be in-store or online</strong>.</p><p>So retailers are constantly focused on ensuring <strong>product availability</strong> and the <strong>right mix of items</strong> for their customers.</p><p>In a previous article, I showed <a href="https://blog.tensorflow.org/2022/05/real-time-sku-detection-in-browser.html">how to create a TensorFlow.js model to recognize SKUs</a>. In this post, we will explore how to <strong>identify empty shelves</strong> using a custom YOLOv7 model — everything running in <strong>real-time, offline, and in the browser of a smartphone</strong>.</p><p>What this post will cover:</p><ul><li>Configuring the environment;</li><li>Gathering the data;</li><li>Preparing the model for training;</li><li>Training and reparametrizing the model;</li><li>Evaluating the model;</li><li>Converting to TensorFlow.js;</li><li>Deploying the model on a web browser.</li></ul><h4>Configuring the environment</h4><p>The entire training pipeline will be executed using the free GPU provided by Google Colab. If you want to access the notebook with all the consolidated steps, <a href="https://colab.research.google.com/drive/185pYr5D2us666KxGim3INaJNAx8vIkcS?usp=sharing">click here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b197596ea3fc6c15a753a2e6266d3c25/href">https://medium.com/media/b197596ea3fc6c15a753a2e6266d3c25/href</a></iframe><h4>Gathering the data</h4><p>The dataset we will use to train the model is the <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910%2FDVN%2F8RET7B&amp;version=DRAFT">Retail empty shelves — stockout</a> (<a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0 1.0 License</a>). It has 3608 annotations from 1155 images and a unique class: <em>Stockout</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qPvwoU3wOtuiCqbUM_t8WA.png" /><figcaption>Dataset sample from <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910%2FDVN%2F8RET7B&amp;version=DRAFT">Harvard Dataverse</a></figcaption></figure><p>The annotations should be in the YOLOv7 format, where each image has its corresponding txt file. The first value in each line represents the class — for the Stockout dataset, all classes are the same and equal to 0. The remaining four numbers in the line indicate the coordinates of the bounding box.</p><blockquote>If you want to create your own dataset, you can use a tool like <a href="https://github.com/opencv/cvat">CVAT</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*z6QGNnd9tMQPWXbFBKsfYA.png" /><figcaption>Example of aYOLOv7 annotation | Image by author</figcaption></figure><p>To download and extract the <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/8RET7B">stockout dataset</a>, use the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccd50986943a7fc813b39f6afa0bfe9a/href">https://medium.com/media/ccd50986943a7fc813b39f6afa0bfe9a/href</a></iframe><h4>Preparing the model for training</h4><p>The first step is to clone the YOLOv7 repository and install the dependencies:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e97398ba0b6fae2bb2a9a47c91dc3d70/href">https://medium.com/media/e97398ba0b6fae2bb2a9a47c91dc3d70/href</a></iframe><p>Then, download the weights pre-trained on the <a href="https://cocodataset.org/#home">COCO 2017 Dataset</a>. They will be used to initialize the model and speed up the training — this technique is known as <a href="https://www.tensorflow.org/tutorials/images/transfer_learning">transfer learning</a>.</p><p>Since we are working on a one-class problem, we opted for YOLOv7-tiny, a lightweight variation of YOLOv7.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d34a77d4f117774f3ae0ff81df37171b/href">https://medium.com/media/d34a77d4f117774f3ae0ff81df37171b/href</a></iframe><p>And before starting the training, we have to configure a .yaml file with the parameters we want to use.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08de2f815b9d744a91f1aa874e033c56/href">https://medium.com/media/08de2f815b9d744a91f1aa874e033c56/href</a></iframe><h4>Training and reparametrizing the model</h4><p>The training process is straightforward and customizable, allowing you to adjust parameters such as the number of epochs and batches to suit your dataset’s requirements.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47b848966300f4f371786a5ecaa7d364/href">https://medium.com/media/47b848966300f4f371786a5ecaa7d364/href</a></iframe><p>By the end of the execution, you will have the weights saved at yolov7/runs/train/your-model-name/weights/best.pt .</p><p>To view the training metrics in a visual format, launch TensorBoard or open the image yolov7/runs/train/your-model-name/results.png .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RFKJoxWDG4FETNJmXRf6cw.png" /><figcaption>Running tensorboard | Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VURY587a8wffDSJPxgQYpw.png" /><figcaption>results.png | Image by author</figcaption></figure><p>Now that you have the model trained, it is time to <strong>reparametrize the weights for inference</strong>.</p><p>Along with its architecture optimizations for real-time object detection, YOLOv7 introduces additional modules and methods that can enhance training efficiency and improve object detection accuracy. These modules, known as <strong>bag-of-freebies</strong>, must be streamlined for efficient inference. For more information, refer to the <a href="https://arxiv.org/abs/2207.02696">model paper</a>.</p><p>Check the weight&#39;s path on the code below and execute it to generate a reparametrized model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccd3a31ff5a58535cc2475aa41000f77/href">https://medium.com/media/ccd3a31ff5a58535cc2475aa41000f77/href</a></iframe><h4>Evaluating the model</h4><p>Now that the model is optimized for inference, it&#39;s time to run it in some testing images to see if it detects empty spaces on the shelves.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4923c36bb22d6fd76ef256ce3dd0c7bd/href">https://medium.com/media/4923c36bb22d6fd76ef256ce3dd0c7bd/href</a></iframe><h4>Converting to TensorFlow.js</h4><p>Converting the model can be challenging, as it requires passing through several transformations: PyTorch to ONNX, ONNX to TensorFlow, and finally, TensorFlow to TensorFlow.js.</p><p>The following code will handle everything for you. Just make sure the <strong>model paths</strong> are correctly configured and then run it!</p><p>Upon completion, the resulting TensorFlow.js model will be automatically downloaded by Google Colab.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/556d4d6d2b45d959477b32e692d14c4d/href">https://medium.com/media/556d4d6d2b45d959477b32e692d14c4d/href</a></iframe><p>Assuming everything has been executed successfully, the model will now be converted to the <strong><em>TensorFlow.js layers format</em></strong>.</p><p>The folder downloaded on your local machine should contain a <em>model.json</em> file and a set of sharded weights files in a binary format. The <em>model.json</em> has both the model topology (aka “architecture” or “graph”: a description of the layers and how they are connected) and a manifest of the weight files.</p><pre>└ stockout_web_model<br>  ├── group1-shard1of6.bin<br>  ├── group1-shard2of6.bin<br>  ├── group1-shard3of6.bin<br>  ├── group1-shard4of6.bin<br>  ├── group1-shard5of6.bin<br>  ├── group1-shard6of6.bin<br>  └── model.json</pre><h4>Deploying the model</h4><p>The model is now ready to be loaded into JavaScript. As mentioned in the beginning, I open-sourced a YOLOv7 code in JavaScript. Therefore, we can utilize the same repository and replace the model with the one that we have just trained</p><p>Clone the <a href="https://github.com/hugozanini/yolov7-tfjs.git">repository</a>:</p><pre>git clone https://github.com/hugozanini/yolov7-tfjs.git</pre><p>Install the packages:</p><pre>npm install</pre><p>Go to the <a href="https://github.com/hugozanini/yolov7-tfjs/tree/master/public">public</a> folder and past the model trained. You should have a structure like this:</p><pre>├── git-media<br>├── index.html<br>├── LICENSE<br>├── node_modules<br>├── package.json<br>├── public<br>│    ├── stockout_web_model<br>│    └── yolov7_web_model<br>├── README.MD<br>└── src</pre><p>Go to <a href="https://github.com/hugozanini/yolov7-tfjs/blob/master/src/App.jsx">src/App.jsx</a> and change the model name on <a href="https://github.com/hugozanini/yolov7-tfjs/blob/4efa4cba39d1168d4bcf7bfe79274e54e87d2eaa/src/App.jsx#L29">line 29</a> to stockout:</p><pre>  const modelName = &quot;stockout&quot;;</pre><p>To execute the application go to the root folder and run the following command:</p><pre>npm start</pre><p>A local server will be started, and you should see something similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*00O2yJGk03CeSA2tcjClvA.gif" /><figcaption>Running example | Image by author</figcaption></figure><p>This model was deployed on CodesSandbox too. Access the link below to see it running.</p><p><a href="https://codesandbox.io/p/github/hugozanini/yolov7-tfjs/stockout?file=%2FREADME.md&amp;workspace=%257B%2522activeFileId%2522%253A%2522clffv5qby000yg1fw2ridgthd%2522%252C%2522openFiles%2522%253A%255B%2522%252FREADME.md%2522%255D%252C%2522sidebarPanel%2522%253A%2522EXPLORER%2522%252C%2522gitSidebarPanel%2522%253A%2522COMMIT%2522%252C%2522spaces%2522%253A%257B%2522clfjm1t8i00733b6idq6lqztm%2522%253A%257B%2522key%2522%253A%2522clfjm1t8i00733b6idq6lqztm%2522%252C%2522name%2522%253A%2522Default%2522%252C%2522devtools%2522%253A%255B%257B%2522type%2522%253A%2522PREVIEW%2522%252C%2522taskId%2522%253A%2522start%2522%252C%2522port%2522%253A3000%252C%2522key%2522%253A%2522clfjm1t8i00743b6iw255o0qk%2522%252C%2522isMinimized%2522%253Afalse%257D%255D%257D%257D%252C%2522currentSpace%2522%253A%2522clfjm1t8i00733b6idq6lqztm%2522%252C%2522spacesOrder%2522%253A%255B%2522clfjm1t8i00733b6idq6lqztm%2522%255D%252C%2522hideCodeEditor%2522%253Afalse%257D">CodeSandbox</a></p><p>With YOLOv7, it’s possible to detect up to 80 different classes. For businesses in the retail industry, this presents a great opportunity to improve their in-store product execution.</p><p>By training the model to recognize all of their company’s products, retailers can ensure that their <strong>products are properly placed on the shelves and that customers are able to easily find what they are looking for</strong>.</p><p>To validate the effectiveness of the model trained, I went to a grocery and drug store with my phone and recorded some examples of the detector running in real time.</p><p>In the video below, you can verify the solution&#39;s ability to accurately detect empty shelves in a real environment.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBuZQbJdZw80%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBuZQbJdZw80&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBuZQbJdZw80%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c08c689e838012a3f74b445050a5d6d/href">https://medium.com/media/5c08c689e838012a3f74b445050a5d6d/href</a></iframe><p>A stockout detector in conjunction with an <a href="https://blog.tensorflow.org/2022/05/real-time-sku-detection-in-browser.html">SKU recognition model</a> can greatly enhance the efficiency and effectiveness of retail operations.</p><p>While cloud-based solutions exist, they can sometimes be slow, taking up to 24 hours to process a single detection. In comparison, using TensorFlow.js models on a smartphone browser allows for offline, real-time recognition — allowing businesses to make more immediate decisions and respond to stockouts faster.</p><p>Overall, combining a stockout detector with an SKU recognition model can provide a powerful way for optimizing retail operations and enhancing the shopping experience for customers. By using real-time analysis and offline recognition capabilities, businesses can make informed decisions and respond quickly to changing conditions.</p><p>If you have any questions or want to share about a user case, feel reach me on <a href="https://www.linkedin.com/in/hugozanini/?locale=en_US">Linkedin</a> or <a href="https://twitter.com/hugoznn">Twitter</a>. All the source code used in this post is available on the<a href="https://github.com/hugozanini/yolov7-tfjs/tree/stockout"> project repo</a>.</p><p>Thanks for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96a5ecd7a530" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/training-a-custom-yolov7-in-pytorch-and-running-it-directly-in-the-browser-with-tensorflow-js-96a5ecd7a530">Training a custom YOLOv7 in PyTorch and running it directly in the browser with TensorFlow.js</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>